Jul 31 14:12:40.005544 ip-10-0-139-143 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 14:12:40.362554 ip-10-0-139-143 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:12:40.362554 ip-10-0-139-143 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 14:12:40.362554 ip-10-0-139-143 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:12:40.362554 ip-10-0-139-143 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 14:12:40.362554 ip-10-0-139-143 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:12:40.363411 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.363293    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 14:12:40.370523 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370508    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:12:40.370523 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370522    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370526    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370530    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370533    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370536    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370538    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370542    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370550    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370553    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370556    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370559    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370561    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370564    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370567    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370569    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370572    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370575    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370578    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370581    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370583    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:12:40.370585 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370586    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370589    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370592    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370595    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370598    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370601    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370603    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370608    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370612    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370615    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370618    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370621    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370624    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370627    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370629    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370632    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370634    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370637    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370639    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370642    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:12:40.371074 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370645    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370648    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370650    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370653    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370656    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370659    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370661    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370664    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370666    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370669    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370672    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370674    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370677    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370679    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370683    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370685    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370688    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370691    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370694    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:12:40.371603 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370698    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370701    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370704    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370707    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370710    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370713    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370716    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370720    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370722    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370725    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370728    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370731    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370735    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370737    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370740    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370742    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370745    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370747    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370750    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370753    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:12:40.372063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370755    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370758    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370761    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370763    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370766    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370768    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370771    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370774    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370777    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370779    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370782    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370785    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370787    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370790    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370792    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370795    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370798    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370801    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.370805    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:12:40.372566 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371501    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371508    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371511    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371515    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371517    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371521    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371523    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371526    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371529    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371531    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371534    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371537    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371541    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371544    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371547    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371550    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371552    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371555    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371557    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:12:40.373030 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371560    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371564    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371568    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371571    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371574    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371577    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371580    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371582    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371585    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371588    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371591    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371594    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371597    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371599    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371602    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371605    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371607    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371610    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371614    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371616    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:12:40.373565 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371619    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371622    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371624    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371627    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371630    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371633    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371636    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371639    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371642    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371645    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371648    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371651    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371653    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371656    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371658    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371661    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371664    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371666    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371669    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371672    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:12:40.374063 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371674    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371676    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371679    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371682    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371685    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371687    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371690    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371693    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371695    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371698    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371700    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371704    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371708    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371711    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371714    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371717    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371721    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371724    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371727    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:12:40.374567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371729    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371732    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371735    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371738    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371740    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371743    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371745    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371748    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371751    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371753    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371756    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371760    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371763    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371766    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371769    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371771    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371774    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371776    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371779    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371782    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:12:40.375048 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.371784    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372131    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372135    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372138    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372141    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372143    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372146    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372149    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372151    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372154    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372157    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372160    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372164    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372167    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372170    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372172    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372175    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372178    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372181    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372183    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:12:40.375552 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372186    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372189    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372192    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372194    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372198    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372202    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372205    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372208    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372210    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372213    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372216    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372219    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372222    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372225    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372228    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372230    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372233    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372235    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372238    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:12:40.376040 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372240    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372244    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372248    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372252    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372255    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372258    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372260    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372263    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372266    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372269    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372271    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372274    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372290    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372293    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372296    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372299    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372303    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372305    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372309    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:12:40.376520 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372312    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372315    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372318    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372320    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372324    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372326    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372329    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372332    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372334    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372338    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372341    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372343    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372346    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372349    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372351    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372357    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372360    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372362    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372366    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372368    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:12:40.376984 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372371    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372373    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372376    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372378    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372381    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372384    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372386    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372389    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372391    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372394    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372397    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372399    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372402    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372405    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372408    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372410    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372413    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372416    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372418    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:12:40.377490 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372421    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372423    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.372426    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372499    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372506    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372513    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372517    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372522    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372527    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372532    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372537    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372540    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372543    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372547    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372550    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372553    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372556    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372559    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372562    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372565    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372568    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372572    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372576    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372579    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 14:12:40.377956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372582    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372585    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372589    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372592    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372596    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372599    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372602    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372606    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372609    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372613    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372615    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372619    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372622    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372625    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372628    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372631    2558 flags.go:64] FLAG: --enable-server="true"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372635    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372641    2558 flags.go:64] FLAG: --event-burst="100"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372644    2558 flags.go:64] FLAG: --event-qps="50"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372647    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372650    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372653    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372657    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372660    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372663    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 14:12:40.378592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372666    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372669    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372672    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372675    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372678    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372681    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372684    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372687    2558 flags.go:64] FLAG: --feature-gates=""
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372691    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372694    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372697    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372701    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372704    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372706    2558 flags.go:64] FLAG: --help="false"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372709    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-139-143.ec2.internal"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372713    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372716    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372719    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372722    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372725    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372728    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372731    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372734    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 14:12:40.379234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372738    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372741    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372744    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372747    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372750    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372753    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372756    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372759    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372762    2558 flags.go:64] FLAG: --lock-file=""
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372765    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372768    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372773    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372778    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372781    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372799    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372823    2558 flags.go:64] FLAG: --logging-format="text"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372828    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372834    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372838    2558 flags.go:64] FLAG: --manifest-url=""
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372841    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372847    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372851    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372856    2558 flags.go:64] FLAG: --max-pods="110"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372860    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372864    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 14:12:40.379862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372867    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372870    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372874    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372877    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372881    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372888    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372891    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372895    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372898    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372901    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372907    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372910    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372914    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372917    2558 flags.go:64] FLAG: --port="10250"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372920    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372923    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b149780261b01445"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372926    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372930    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372933    2558 flags.go:64] FLAG: --register-node="true"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372936    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372940    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372943    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372947    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372951    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372956    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 14:12:40.380842 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372959    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372963    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372966    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372969    2558 flags.go:64] FLAG: --runonce="false"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372972    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372975    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372979    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372982    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372985    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372988    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372991    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372994    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.372997    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373000    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373004    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373008    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373011    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373015    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373018    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373024    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373026    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373030    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373035    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373039    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373042    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 14:12:40.381600 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373045    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373048    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373051    2558 flags.go:64] FLAG: --v="2"
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373055    2558 flags.go:64] FLAG: --version="false"
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373060    2558 flags.go:64] FLAG: --vmodule=""
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373064    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.373068    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373211    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373216    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373219    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373222    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373225    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373227    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373230    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373233    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373237    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373241    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373244    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373247    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373250    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373253    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:12:40.382257 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373258    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373261    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373264    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373267    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373270    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373273    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373288    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373291    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373295    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373299    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373303    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373307    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373310    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373313    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373316    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373318    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373323    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373326    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373329    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:12:40.382788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373332    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373335    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373338    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373340    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373343    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373346    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373349    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373351    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373354    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373356    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373359    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373362    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373365    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373368    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373371    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373374    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373377    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373379    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373382    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373385    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:12:40.383266 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373387    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373390    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373393    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373395    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373398    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373401    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373403    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373406    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373409    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373412    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373415    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373418    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373421    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373424    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373427    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373430    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373432    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373435    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373438    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373440    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:12:40.383775 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373443    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373446    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373448    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373451    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373453    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373458    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373460    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373463    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373465    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373468    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373471    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373474    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373476    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373479    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373482    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373484    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373486    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373489    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373492    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373495    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:12:40.384273 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373497    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373502    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373504    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373507    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373510    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.373513    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.374290    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.382762    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.382781    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.383508    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.383520    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383578    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383584    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383587    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383590    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:12:40.384794 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383593    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383597    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383600    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383604    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383607    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383610    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383613    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383616    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383618    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383621    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383624    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383626    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383629    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383632    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383634    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383639    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383643    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383646    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383649    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383660    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:12:40.385169 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383663    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383666    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383668    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383671    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383674    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383677    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383680    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383683    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383686    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383689    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383692    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383694    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383697    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383700    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383702    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383705    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383708    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383710    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383713    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383716    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:12:40.385698 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383719    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383721    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383724    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383727    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383730    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383732    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383735    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383737    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383740    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383742    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383745    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383747    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383750    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383753    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383755    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383757    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383760    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383764    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383766    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383769    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:12:40.386193 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383772    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383774    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383777    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383780    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383783    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383785    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383788    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383791    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383794    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383796    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383799    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383802    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383806    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383809    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383812    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383815    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383818    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383820    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383823    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:12:40.386804 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383826    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383828    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383832    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383835    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383838    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383840    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383843    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383847    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383851    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383854    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383857    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383860    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383862    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383865    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383868    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:12:40.387301 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383870    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.383876    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383993    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.383998    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384001    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384005    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384009    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384012    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384015    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384018    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384021    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384024    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384027    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384030    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384033    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:12:40.387679 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384036    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384039    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384042    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384044    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384047    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384050    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384053    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384055    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384058    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384061    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384063    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384066    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384069    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384072    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384074    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384077    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384080    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384082    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384084    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384088    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:12:40.388058 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384091    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384095    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384098    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384100    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384103    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384106    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384108    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384111    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384114    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384117    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384120    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384122    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384126    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384129    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384131    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384134    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384136    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384139    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384142    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:12:40.388567 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384144    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384147    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384149    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384153    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384156    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384159    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384163    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384166    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384168    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384171    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384173    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384176    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384179    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384181    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384184    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384186    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384189    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384191    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384194    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384196    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:12:40.389025 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384199    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384202    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384204    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384207    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384209    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384212    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384215    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384218    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384220    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384223    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384226    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384228    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384231    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384234    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384236    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384239    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384242    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384244    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384247    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384249    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:12:40.389636 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384252    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384255    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384257    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384260    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384263    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384265    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:40.384268    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.384273    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.384412    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.387062    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.388327    2558 server.go:1019] "Starting client certificate rotation"
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.388423    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:12:40.390111 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.388456    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:12:40.390620 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.390607    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:12:40.394936 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.394918    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:12:40.404760 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.404742    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 31 14:12:40.406754 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.406739    2558 log.go:25] "Validated CRI v1 image API"
Jul 31 14:12:40.407598 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.407567    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 14:12:40.409582 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.409562    2558 fs.go:135] Filesystem UUIDs: map[092b20c6-70c6-4262-8b32-6af31589cb5f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 91e549dd-79d1-4328-acf7-26ca6a8d8f23:/dev/nvme0n1p4]
Jul 31 14:12:40.409657 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.409581    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 14:12:40.415751 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.415635    2558 manager.go:216] Machine: {Timestamp:2026-07-31 14:12:40.413952149 +0000 UTC m=+0.303260361 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100917 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec200eacdf975a091a971deac95a730b SystemUUID:ec200eac-df97-5a09-1a97-1deac95a730b BootID:ed0672bc-cf2a-44db-aa86-006e130cfc23 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f9:ad:10:fe:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f9:ad:10:fe:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:ad:a1:27:98:28 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 14:12:40.415751 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.415742    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 14:12:40.415928 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.415858    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 14:12:40.417649 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.417624    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 14:12:40.417776 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.417651    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-143.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 14:12:40.418695 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.418685    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 14:12:40.418735 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.418697    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 14:12:40.418735 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.418706    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:12:40.419291 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.419269    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:12:40.420183 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.420173    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 14:12:40.420313 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.420296    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:12:40.423340 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423327    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:12:40.423456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423446    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 14:12:40.423490 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423476    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 14:12:40.423490 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423485    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 14:12:40.423545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423497    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 14:12:40.423545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423505    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 31 14:12:40.423545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423514    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 14:12:40.423860 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423846    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:12:40.423910 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.423865    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:12:40.425934 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.425911    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 14:12:40.427170 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.427157    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 14:12:40.427212 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.427186    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 14:12:40.428105 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428094    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 14:12:40.428145 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428109    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 14:12:40.428145 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428115    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 14:12:40.428145 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428121    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 14:12:40.428145 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428131    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 14:12:40.428145 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428137    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 14:12:40.428145 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428147    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 14:12:40.428321 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428152    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 14:12:40.428321 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428159    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 14:12:40.428321 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428164    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 14:12:40.428321 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428173    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 14:12:40.428321 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428179    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 14:12:40.428862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428853    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 14:12:40.428862 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.428861    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 14:12:40.431904 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.431891    2558 server.go:1268] "Started kubelet"
Jul 31 14:12:40.432040 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.431984    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 14:12:40.432096 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.432068    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 14:12:40.432347 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.432313    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 14:12:40.432688 ip-10-0-139-143 systemd[1]: Started Kubernetes Kubelet.
Jul 31 14:12:40.433081 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.433060    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 14:12:40.433862 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.433804    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 14:12:40.433919 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.433785    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-143.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 14:12:40.433919 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.433888    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 14:12:40.434273 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.434257    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 14:12:40.437391 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.436615    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-143.ec2.internal.18c7654d69752f39  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-143.ec2.internal,UID:ip-10-0-139-143.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-143.ec2.internal,},FirstTimestamp:2026-07-31 14:12:40.431865657 +0000 UTC m=+0.321173869,LastTimestamp:2026-07-31 14:12:40.431865657 +0000 UTC m=+0.321173869,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-143.ec2.internal,}"
Jul 31 14:12:40.438941 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.438927    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 14:12:40.438983 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.438950    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 14:12:40.439039 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.439023    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:40.439445 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.439424    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 14:12:40.439445 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.439440    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 14:12:40.439561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.439423    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 14:12:40.439561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.439528    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 14:12:40.439561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.439536    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 14:12:40.440256 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.440226    2558 factory.go:55] Registering systemd factory
Jul 31 14:12:40.440256 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.440246    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 31 14:12:40.441887 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.441773    2558 factory.go:153] Registering CRI-O factory
Jul 31 14:12:40.441887 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.441791    2558 factory.go:223] Registration of the crio container factory successfully
Jul 31 14:12:40.441887 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.441837    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 14:12:40.441887 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.441857    2558 factory.go:103] Registering Raw factory
Jul 31 14:12:40.441887 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.441871    2558 manager.go:1196] Started watching for new ooms in manager
Jul 31 14:12:40.442598 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.442219    2558 manager.go:349] Starting recovery of all containers
Jul 31 14:12:40.443515 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.443492    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 14:12:40.445004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.444980    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dkh9k"
Jul 31 14:12:40.451034 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.451004    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 14:12:40.451130 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.451111    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-143.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 14:12:40.453868 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.453845    2558 manager.go:354] Recovery completed
Jul 31 14:12:40.453943 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.453908    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dkh9k"
Jul 31 14:12:40.458210 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.458196    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:12:40.460174 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.460157    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:12:40.460234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.460191    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:12:40.460234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.460204    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:12:40.460778 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.460760    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 14:12:40.460841 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.460778    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 14:12:40.460841 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.460790    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:12:40.460841 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.460796    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:12:40.461424 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.461412    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:12:40.461469 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.461440    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:12:40.461469 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.461454    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:12:40.461957 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.461945    2558 policy_none.go:49] "None policy: Start"
Jul 31 14:12:40.461987 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.461962    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 14:12:40.461987 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.461974    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 14:12:40.462896 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.462878    2558 policy_none.go:47] "Start"
Jul 31 14:12:40.513927 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.513768    2558 manager.go:337] "Starting Device Plugin manager"
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.513941    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.513956    2558 server.go:87] "Starting device plugin registration server"
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.514195    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.514205    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.514343    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.514433    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.514441    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.515449    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 14:12:40.529443 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.515501    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:40.554660 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.554628    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 14:12:40.555855 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.555840    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 14:12:40.555934 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.555888    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 14:12:40.555934 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.555907    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 14:12:40.556024 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:40.555938    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 14:12:40.558152 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:40.558133    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:12:41.631847 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.631778    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:12:41.632784 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.632767    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:12:41.632850 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.632798    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:12:41.632850 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.632823    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:12:41.632912 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.632862    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.640936 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.640920    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.640996 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:41.640942    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-143.ec2.internal\": node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:41.657289 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:41.657264    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:41.673783 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.673767    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"]
Jul 31 14:12:41.673857 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.673824    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:12:41.675186 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.675171    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:12:41.675280 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.675197    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:12:41.675280 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.675220    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:12:41.676386 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.676374    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:12:41.676566 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.676551    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.676618 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.676582    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:12:41.677123 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.677107    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:12:41.677224 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.677110    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:12:41.677224 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.677161    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:12:41.677224 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.677178    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:12:41.677224 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.677134    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:12:41.677224 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.677206    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:12:41.678211 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.678192    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.678284 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.678224    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:12:41.678858 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.678846    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:12:41.678924 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.678865    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:12:41.678924 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.678875    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:12:41.699609 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:41.699588    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-143.ec2.internal\" not found" node="ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.703742 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:41.703726    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-143.ec2.internal\" not found" node="ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.757337 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.757299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fcb5a47836661934246ad400500ba6e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-143.ec2.internal\" (UID: \"0fcb5a47836661934246ad400500ba6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.757337 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:41.757316    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:41.757527 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.757343    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d443483119bcac6f2b2616712ae7bca3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"d443483119bcac6f2b2616712ae7bca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.757527 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.757377    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d443483119bcac6f2b2616712ae7bca3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"d443483119bcac6f2b2616712ae7bca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.858401 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:41.858358    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:41.858401 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.858386    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fcb5a47836661934246ad400500ba6e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-143.ec2.internal\" (UID: \"0fcb5a47836661934246ad400500ba6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.858605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.858421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d443483119bcac6f2b2616712ae7bca3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"d443483119bcac6f2b2616712ae7bca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.858605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.858455    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d443483119bcac6f2b2616712ae7bca3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"d443483119bcac6f2b2616712ae7bca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.858605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.858479    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d443483119bcac6f2b2616712ae7bca3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"d443483119bcac6f2b2616712ae7bca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.858605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.858483    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0fcb5a47836661934246ad400500ba6e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-143.ec2.internal\" (UID: \"0fcb5a47836661934246ad400500ba6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.858605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:41.858483    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d443483119bcac6f2b2616712ae7bca3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal\" (UID: \"d443483119bcac6f2b2616712ae7bca3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:41.959071 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:41.958987    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.002191 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.002167    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:42.006393 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.006127    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:42.059879 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.059847    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.160335 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.160306    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.260734 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.260672    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.361082 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.361051    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.404252 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.404227    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 14:12:42.404657 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.404369    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:12:42.461859 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.461829    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.475292 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.475234    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:07:41 +0000 UTC" deadline="2028-05-10 10:11:00.37949727 +0000 UTC"
Jul 31 14:12:42.475292 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.475274    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:12:42.475490 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.475276    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15571h58m17.904225694s"
Jul 31 14:12:42.532556 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:42.532523    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fcb5a47836661934246ad400500ba6e.slice/crio-feb43f02b7adb1490a0a38b9b831b86d4d32867746a249f0068e62c287a4f515 WatchSource:0}: Error finding container feb43f02b7adb1490a0a38b9b831b86d4d32867746a249f0068e62c287a4f515: Status 404 returned error can't find the container with id feb43f02b7adb1490a0a38b9b831b86d4d32867746a249f0068e62c287a4f515
Jul 31 14:12:42.532820 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:42.532798    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd443483119bcac6f2b2616712ae7bca3.slice/crio-be832418bf8c0a3f830358d9e220a4a32bb60b538bb6e20557c9c3adc6f0a901 WatchSource:0}: Error finding container be832418bf8c0a3f830358d9e220a4a32bb60b538bb6e20557c9c3adc6f0a901: Status 404 returned error can't find the container with id be832418bf8c0a3f830358d9e220a4a32bb60b538bb6e20557c9c3adc6f0a901
Jul 31 14:12:42.537750 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.537736    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:12:42.562669 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.562645    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.575697 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.575654    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal" event={"ID":"d443483119bcac6f2b2616712ae7bca3","Type":"ContainerStarted","Data":"be832418bf8c0a3f830358d9e220a4a32bb60b538bb6e20557c9c3adc6f0a901"}
Jul 31 14:12:42.576631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.576608    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal" event={"ID":"0fcb5a47836661934246ad400500ba6e","Type":"ContainerStarted","Data":"feb43f02b7adb1490a0a38b9b831b86d4d32867746a249f0068e62c287a4f515"}
Jul 31 14:12:42.662996 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.662961    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.763435 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.763398    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.777288 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.777217    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:12:42.863685 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:42.863647    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-143.ec2.internal\" not found"
Jul 31 14:12:42.933529 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.933502    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:12:42.956924 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.956904    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:42.967924 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.967902    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:12:42.969009 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.968991    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal"
Jul 31 14:12:42.977204 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:42.977191    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:12:43.441298 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.441266    2558 apiserver.go:52] "Watching apiserver"
Jul 31 14:12:43.449589 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.449553    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 14:12:43.450106 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.450084    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-rtc4t","openshift-image-registry/node-ca-gkw4n","openshift-multus/multus-wccvh","openshift-network-diagnostics/network-check-target-xxbnh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal","openshift-multus/multus-additional-cni-plugins-wp2xh","openshift-multus/network-metrics-daemon-jbbns","openshift-network-operator/iptables-alerter-hlrmr","openshift-ovn-kubernetes/ovnkube-node-zvkmj","kube-system/global-pull-secret-syncer-sfw5j","kube-system/konnectivity-agent-zlw8q","kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tx8nv"]
Jul 31 14:12:43.451583 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.451561    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.452769 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.452749    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.452855 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.452817    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:43.453995 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.453975    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.454620 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.454602    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 14:12:43.454815 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.454794    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 14:12:43.454910 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.454892    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 14:12:43.455183 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.455161    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:43.455269 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.455242    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:12:43.455328 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.455269    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 14:12:43.455328 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.455309    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fk7r5\""
Jul 31 14:12:43.455453 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.455339    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 14:12:43.456710 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.456692    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 14:12:43.456863 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.456848    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 14:12:43.456940 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.456887    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sr4l2\""
Jul 31 14:12:43.457255 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.457238    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.458414 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.458396    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.460630 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.460609    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.460715 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.460702    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 14:12:43.461016 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.460997    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 14:12:43.461347 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.461218    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fg966\""
Jul 31 14:12:43.461347 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.461230    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:12:43.461347 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.461245    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 14:12:43.461347 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.461341    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6rkgr\""
Jul 31 14:12:43.461863 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.461831    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:43.461943 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.461891    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 14:12:43.461943 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.461903    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:43.463761 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.463391    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.464260 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.464108    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:12:43.464451 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.464434    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jt8kf\""
Jul 31 14:12:43.464804 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.464790    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.467418 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.466861    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.467614    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-system-cni-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.467664    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.467676    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-socket-dir-parent\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.467734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-conf-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.467764    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.467888    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-registration-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.467958    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-646fl\""
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-cni-binary-copy\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468220    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468291    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a778adb-72c6-427b-97e3-9da9685b0467-kubelet-config\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-cni-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468410    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-socket-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468468    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468482    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxpfb\""
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468611    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-cnibin\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468675    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7gm\" (UniqueName: \"kubernetes.io/projected/37ea6151-ad56-41c1-8ed5-ede42ab94632-kube-api-access-jv7gm\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.469357 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-k8s-cni-cncf-io\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.470343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468774    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvgf\" (UniqueName: \"kubernetes.io/projected/83810d6e-f7ae-46d0-a38c-4bab260b035d-kube-api-access-4cvgf\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.470343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468810    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-sys-fs\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.470343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468834    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 14:12:43.470343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468844    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a778adb-72c6-427b-97e3-9da9685b0467-dbus\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.470343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468878    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 14:12:43.470343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.468921    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83810d6e-f7ae-46d0-a38c-4bab260b035d-cni-binary-copy\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.470343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.469459    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-kubelet\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.470343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.470097    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 14:12:43.470702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.470469    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 14:12:43.470747 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.470729    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 14:12:43.471129 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.471112    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 14:12:43.471680 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.471663    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 14:12:43.471805 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.469691    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-multus-certs\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.471877 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.471838    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:43.471877 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.471866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.471982 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.471891    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.472235 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.471889    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-cnibin\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.472316 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-cni-bin\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.472316 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472287    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-hostroot\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.472427 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472340    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-etc-kubernetes\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.472427 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-etc-selinux\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.472427 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472408    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-os-release\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.472574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472432    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-daemon-config\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.472574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-os-release\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.472574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472477    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.472574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472500    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-device-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.472574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472524    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.472574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-netns\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.472574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472573    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-cni-multus\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.472890 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.472598    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmvn8\" (UniqueName: \"kubernetes.io/projected/2ba92576-73a7-4597-964d-d18042bd4228-kube-api-access-fmvn8\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.473535 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.473518    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:12:43.474543 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.474520    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:12:43.474636 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.474574    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kv7hd\""
Jul 31 14:12:43.474795 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.474781    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 14:12:43.474886 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.474845    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 14:12:43.476404 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.476386    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 14:12:43.476608 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.476568    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2srmt\""
Jul 31 14:12:43.477438 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.477281    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 14:12:43.494657 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.494636    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dwqq7"
Jul 31 14:12:43.504074 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.504053    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dwqq7"
Jul 31 14:12:43.557742 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.557713    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 14:12:43.567828 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.567797    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:12:43.573123 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573096    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a778adb-72c6-427b-97e3-9da9685b0467-kubelet-config\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.573226 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-cni-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573226 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573170    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c7c983-8b4d-4494-be24-c63071b5ad0a-hosts-file\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.573226 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573212    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-cnibin\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1a778adb-72c6-427b-97e3-9da9685b0467-kubelet-config\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573236    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-k8s-cni-cncf-io\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvgf\" (UniqueName: \"kubernetes.io/projected/83810d6e-f7ae-46d0-a38c-4bab260b035d-kube-api-access-4cvgf\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573308    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-run\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573330    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84db4ce3-b047-40a8-a504-1bd5651ffba0-tmp\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573338    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-cnibin\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d76362e-39cc-4a7a-88df-8902bf386dc1-konnectivity-ca\") pod \"konnectivity-agent-zlw8q\" (UID: \"3d76362e-39cc-4a7a-88df-8902bf386dc1\") " pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-sys-fs\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.573396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a778adb-72c6-427b-97e3-9da9685b0467-dbus\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573418    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83810d6e-f7ae-46d0-a38c-4bab260b035d-cni-binary-copy\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-cni-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-kubelet\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573526    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-lib-modules\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c7c983-8b4d-4494-be24-c63071b5ad0a-tmp-dir\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-cnibin\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573607    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-cni-bin\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-etc-kubernetes\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-daemon-config\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573677    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-cni-bin\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573283    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-k8s-cni-cncf-io\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573703    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovnkube-config\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573727    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-os-release\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573753    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66f05e4e-0197-4037-923c-c626b7982ab2-host\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.573800 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-sys-fs\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573836    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-cnibin\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573873    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-kubelet\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.573775    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-node-log\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-cni-netd\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovnkube-script-lib\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysconfig\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-cni-bin\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574216    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-etc-kubernetes\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-os-release\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d76362e-39cc-4a7a-88df-8902bf386dc1-agent-certs\") pod \"konnectivity-agent-zlw8q\" (UID: \"3d76362e-39cc-4a7a-88df-8902bf386dc1\") " pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574289    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574336    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmvn8\" (UniqueName: \"kubernetes.io/projected/2ba92576-73a7-4597-964d-d18042bd4228-kube-api-access-fmvn8\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.574363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574364    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-system-cni-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-conf-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574414    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-systemd-units\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574446    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574471    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-modprobe-d\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574504    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysctl-d\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shcf4\" (UniqueName: \"kubernetes.io/projected/84db4ce3-b047-40a8-a504-1bd5651ffba0-kube-api-access-shcf4\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574555    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-registration-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1a778adb-72c6-427b-97e3-9da9685b0467-dbus\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574622    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-system-cni-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-registration-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574695    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-conf-dir\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.574759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574726    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-cni-binary-copy\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqxx\" (UniqueName: \"kubernetes.io/projected/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-kube-api-access-kzqxx\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574880    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-slash\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574922    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-tuned\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-socket-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.574991    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/83810d6e-f7ae-46d0-a38c-4bab260b035d-cni-binary-copy\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575014    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7gm\" (UniqueName: \"kubernetes.io/projected/37ea6151-ad56-41c1-8ed5-ede42ab94632-kube-api-access-jv7gm\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-iptables-alerter-script\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-run-netns\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575097    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-system-cni-dir\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575119    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-env-overrides\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.575178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575145    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-systemd\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575187    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575201    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-socket-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-sys\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-multus-certs\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575311    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575360    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-daemon-config\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-log-socket\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575395    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-multus-certs\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575433    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r22c5\" (UniqueName: \"kubernetes.io/projected/bb0c96e2-5e54-454f-b865-a7ee847852d2-kube-api-access-r22c5\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575466    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qch4n\" (UniqueName: \"kubernetes.io/projected/57c7c983-8b4d-4494-be24-c63071b5ad0a-kube-api-access-qch4n\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575558    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-hostroot\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-host-slash\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-kubelet\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-cni-binary-copy\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.575808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-var-lib-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575824    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-var-lib-kubelet\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-hostroot\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575865    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-etc-selinux\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-os-release\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.575970    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-os-release\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-etc-selinux\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576051    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37ea6151-ad56-41c1-8ed5-ede42ab94632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovn-node-metrics-cert\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576099    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-host\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576125    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgrbj\" (UniqueName: \"kubernetes.io/projected/66f05e4e-0197-4037-923c-c626b7982ab2-kube-api-access-fgrbj\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-kubernetes\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576210    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-device-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-netns\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.576268    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576272    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-cni-multus\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576307    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66f05e4e-0197-4037-923c-c626b7982ab2-serviceca\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.576561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576332    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-systemd\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-socket-dir-parent\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktww8\" (UniqueName: \"kubernetes.io/projected/13d90876-b172-48d8-95e3-420e703c71cf-kube-api-access-ktww8\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576449    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-etc-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576483    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-ovn\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysctl-conf\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576540    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576566    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576831    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37ea6151-ad56-41c1-8ed5-ede42ab94632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576881    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-device-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.576881    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret podName:1a778adb-72c6-427b-97e3-9da9685b0467 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:44.076858895 +0000 UTC m=+2.949422140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret") pod "global-pull-secret-syncer-sfw5j" (UID: "1a778adb-72c6-427b-97e3-9da9685b0467") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-run-netns\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-multus-socket-dir-parent\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.576962    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/83810d6e-f7ae-46d0-a38c-4bab260b035d-host-var-lib-cni-multus\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.577350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.577049    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ba92576-73a7-4597-964d-d18042bd4228-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.579249 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.579109    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:12:43.579249 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.579135    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:12:43.579249 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.579148    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pjj9m for pod openshift-network-diagnostics/network-check-target-xxbnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:43.579249 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.579228    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m podName:6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f nodeName:}" failed. No retries permitted until 2026-07-31 14:12:44.079200923 +0000 UTC m=+2.951764174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pjj9m" (UniqueName: "kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m") pod "network-check-target-xxbnh" (UID: "6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:43.579249 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.579234    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 14:12:43.583265 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.583247    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmvn8\" (UniqueName: \"kubernetes.io/projected/2ba92576-73a7-4597-964d-d18042bd4228-kube-api-access-fmvn8\") pod \"aws-ebs-csi-driver-node-6j55x\" (UID: \"2ba92576-73a7-4597-964d-d18042bd4228\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.583369 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.583294    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvgf\" (UniqueName: \"kubernetes.io/projected/83810d6e-f7ae-46d0-a38c-4bab260b035d-kube-api-access-4cvgf\") pod \"multus-wccvh\" (UID: \"83810d6e-f7ae-46d0-a38c-4bab260b035d\") " pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.583369 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.583301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7gm\" (UniqueName: \"kubernetes.io/projected/37ea6151-ad56-41c1-8ed5-ede42ab94632-kube-api-access-jv7gm\") pod \"multus-additional-cni-plugins-wp2xh\" (UID: \"37ea6151-ad56-41c1-8ed5-ede42ab94632\") " pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.677438 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgrbj\" (UniqueName: \"kubernetes.io/projected/66f05e4e-0197-4037-923c-c626b7982ab2-kube-api-access-fgrbj\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.677438 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-kubernetes\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.677667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677483    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66f05e4e-0197-4037-923c-c626b7982ab2-serviceca\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.677667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677508    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-kubernetes\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.677667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-systemd\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.677667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677598    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-systemd\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.677667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktww8\" (UniqueName: \"kubernetes.io/projected/13d90876-b172-48d8-95e3-420e703c71cf-kube-api-access-ktww8\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:43.677667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-etc-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677683    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-ovn\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677706    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysctl-conf\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677735    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c7c983-8b4d-4494-be24-c63071b5ad0a-hosts-file\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677760    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-run\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677781    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84db4ce3-b047-40a8-a504-1bd5651ffba0-tmp\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d76362e-39cc-4a7a-88df-8902bf386dc1-konnectivity-ca\") pod \"konnectivity-agent-zlw8q\" (UID: \"3d76362e-39cc-4a7a-88df-8902bf386dc1\") " pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677849    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677872    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-lib-modules\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677895    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c7c983-8b4d-4494-be24-c63071b5ad0a-tmp-dir\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-cni-bin\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.677944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovnkube-config\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677969    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66f05e4e-0197-4037-923c-c626b7982ab2-host\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.677990    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-node-log\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678013    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-cni-netd\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovnkube-script-lib\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysconfig\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d76362e-39cc-4a7a-88df-8902bf386dc1-agent-certs\") pod \"konnectivity-agent-zlw8q\" (UID: \"3d76362e-39cc-4a7a-88df-8902bf386dc1\") " pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-systemd-units\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678160    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-modprobe-d\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678208    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysctl-d\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678232    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shcf4\" (UniqueName: \"kubernetes.io/projected/84db4ce3-b047-40a8-a504-1bd5651ffba0-kube-api-access-shcf4\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqxx\" (UniqueName: \"kubernetes.io/projected/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-kube-api-access-kzqxx\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678263    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66f05e4e-0197-4037-923c-c626b7982ab2-serviceca\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-slash\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678306    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678330    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-tuned\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.678457 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-iptables-alerter-script\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-run-netns\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-env-overrides\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678439    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-systemd\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-sys\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-log-socket\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678533    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r22c5\" (UniqueName: \"kubernetes.io/projected/bb0c96e2-5e54-454f-b865-a7ee847852d2-kube-api-access-r22c5\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678559    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qch4n\" (UniqueName: \"kubernetes.io/projected/57c7c983-8b4d-4494-be24-c63071b5ad0a-kube-api-access-qch4n\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678588    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-host-slash\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678612    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-kubelet\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678636    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-var-lib-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-var-lib-kubelet\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678688    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovn-node-metrics-cert\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-host\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678809    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-host\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-systemd-units\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.678896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-modprobe-d\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679009    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679068    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-sys\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679072    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-systemd\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679124    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-log-socket\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679173    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysctl-d\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.679236    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:43.679296    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:12:44.179279285 +0000 UTC m=+3.051842531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679293    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57c7c983-8b4d-4494-be24-c63071b5ad0a-hosts-file\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679343    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-etc-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-run-ovn\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679381    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-host-slash\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679416    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-kubelet\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679461    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-var-lib-openvswitch\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysctl-conf\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679533    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-run-netns\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679590    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-run\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.679998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679729    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66f05e4e-0197-4037-923c-c626b7982ab2-host\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.679846    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-lib-modules\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-env-overrides\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680112    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-var-lib-kubelet\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57c7c983-8b4d-4494-be24-c63071b5ad0a-tmp-dir\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-cni-bin\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680325    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3d76362e-39cc-4a7a-88df-8902bf386dc1-konnectivity-ca\") pod \"konnectivity-agent-zlw8q\" (UID: \"3d76362e-39cc-4a7a-88df-8902bf386dc1\") " pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680385    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-run-ovn-kubernetes\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-cni-netd\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-node-log\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680712    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovnkube-config\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-sysconfig\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.680913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.680827    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb0c96e2-5e54-454f-b865-a7ee847852d2-host-slash\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.681487 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.681396    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-iptables-alerter-script\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.682126 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.682058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3d76362e-39cc-4a7a-88df-8902bf386dc1-agent-certs\") pod \"konnectivity-agent-zlw8q\" (UID: \"3d76362e-39cc-4a7a-88df-8902bf386dc1\") " pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:12:43.682687 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.682475    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovn-node-metrics-cert\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.682687 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.682648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/84db4ce3-b047-40a8-a504-1bd5651ffba0-etc-tuned\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.683421 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.683400    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84db4ce3-b047-40a8-a504-1bd5651ffba0-tmp\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.686410 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.686389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgrbj\" (UniqueName: \"kubernetes.io/projected/66f05e4e-0197-4037-923c-c626b7982ab2-kube-api-access-fgrbj\") pod \"node-ca-gkw4n\" (UID: \"66f05e4e-0197-4037-923c-c626b7982ab2\") " pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.687066 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.687007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktww8\" (UniqueName: \"kubernetes.io/projected/13d90876-b172-48d8-95e3-420e703c71cf-kube-api-access-ktww8\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:43.688102 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.688067    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqxx\" (UniqueName: \"kubernetes.io/projected/8713ceba-51f1-4cf6-8ea4-4768ebce9cd4-kube-api-access-kzqxx\") pod \"iptables-alerter-hlrmr\" (UID: \"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4\") " pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.688897 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.688877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r22c5\" (UniqueName: \"kubernetes.io/projected/bb0c96e2-5e54-454f-b865-a7ee847852d2-kube-api-access-r22c5\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.689319 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.689291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qch4n\" (UniqueName: \"kubernetes.io/projected/57c7c983-8b4d-4494-be24-c63071b5ad0a-kube-api-access-qch4n\") pod \"node-resolver-rtc4t\" (UID: \"57c7c983-8b4d-4494-be24-c63071b5ad0a\") " pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.689385 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.689355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shcf4\" (UniqueName: \"kubernetes.io/projected/84db4ce3-b047-40a8-a504-1bd5651ffba0-kube-api-access-shcf4\") pod \"tuned-tx8nv\" (UID: \"84db4ce3-b047-40a8-a504-1bd5651ffba0\") " pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.690444 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.690427    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb0c96e2-5e54-454f-b865-a7ee847852d2-ovnkube-script-lib\") pod \"ovnkube-node-zvkmj\" (UID: \"bb0c96e2-5e54-454f-b865-a7ee847852d2\") " pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.763556 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.763488    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wp2xh"
Jul 31 14:12:43.773796 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.773767    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wccvh"
Jul 31 14:12:43.785101 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.785081    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x"
Jul 31 14:12:43.791340 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.791321    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tx8nv"
Jul 31 14:12:43.798584 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.798567    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hlrmr"
Jul 31 14:12:43.805873 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.805857    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gkw4n"
Jul 31 14:12:43.813107 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.813090    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:12:43.820317 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.820298    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rtc4t"
Jul 31 14:12:43.825702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:43.825580    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:12:44.081374 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.081286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:44.081374 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.081357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:44.081570 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.081439    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:44.081570 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.081492    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:12:44.081570 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.081509    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:12:44.081570 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.081520    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret podName:1a778adb-72c6-427b-97e3-9da9685b0467 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:45.081493805 +0000 UTC m=+3.954057047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret") pod "global-pull-secret-syncer-sfw5j" (UID: "1a778adb-72c6-427b-97e3-9da9685b0467") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:44.081570 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.081522    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pjj9m for pod openshift-network-diagnostics/network-check-target-xxbnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:44.081772 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.081576    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m podName:6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f nodeName:}" failed. No retries permitted until 2026-07-31 14:12:45.081558789 +0000 UTC m=+3.954122021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pjj9m" (UniqueName: "kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m") pod "network-check-target-xxbnh" (UID: "6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:44.181893 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.181857    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:44.182073 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.182053    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:44.182145 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.182132    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:12:45.182111489 +0000 UTC m=+4.054674726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:44.502374 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:44.502281    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84db4ce3_b047_40a8_a504_1bd5651ffba0.slice/crio-d4aedf707a14cabda2da32e5ff3486891e57c1098b9e040a331fb7419e7ea42f WatchSource:0}: Error finding container d4aedf707a14cabda2da32e5ff3486891e57c1098b9e040a331fb7419e7ea42f: Status 404 returned error can't find the container with id d4aedf707a14cabda2da32e5ff3486891e57c1098b9e040a331fb7419e7ea42f
Jul 31 14:12:44.504420 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:44.504392    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83810d6e_f7ae_46d0_a38c_4bab260b035d.slice/crio-0d4c6ee59dc14367a6d68c6429b68fa69900af3b01211fe4b54ab53322712841 WatchSource:0}: Error finding container 0d4c6ee59dc14367a6d68c6429b68fa69900af3b01211fe4b54ab53322712841: Status 404 returned error can't find the container with id 0d4c6ee59dc14367a6d68c6429b68fa69900af3b01211fe4b54ab53322712841
Jul 31 14:12:44.505168 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:44.505145    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66f05e4e_0197_4037_923c_c626b7982ab2.slice/crio-45da7f53e5738a5e591178d2accc5cc1a3dafddb0bba3cbbbeb2fc4901590885 WatchSource:0}: Error finding container 45da7f53e5738a5e591178d2accc5cc1a3dafddb0bba3cbbbeb2fc4901590885: Status 404 returned error can't find the container with id 45da7f53e5738a5e591178d2accc5cc1a3dafddb0bba3cbbbeb2fc4901590885
Jul 31 14:12:44.506488 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.506295    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:07:43 +0000 UTC" deadline="2028-03-09 23:48:02.197025066 +0000 UTC"
Jul 31 14:12:44.506488 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.506322    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14097h35m17.69070719s"
Jul 31 14:12:44.508443 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:44.508420    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0c96e2_5e54_454f_b865_a7ee847852d2.slice/crio-7d1cdfb5c91d0612a62a9a5048df375fd53d95c4a794e9a330076be60e1b9bd8 WatchSource:0}: Error finding container 7d1cdfb5c91d0612a62a9a5048df375fd53d95c4a794e9a330076be60e1b9bd8: Status 404 returned error can't find the container with id 7d1cdfb5c91d0612a62a9a5048df375fd53d95c4a794e9a330076be60e1b9bd8
Jul 31 14:12:44.509654 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:44.509632    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c7c983_8b4d_4494_be24_c63071b5ad0a.slice/crio-371d9cc10c6035af6eb27395ff12d8a58978e6a776f179a4c5b48a5f75b0e62c WatchSource:0}: Error finding container 371d9cc10c6035af6eb27395ff12d8a58978e6a776f179a4c5b48a5f75b0e62c: Status 404 returned error can't find the container with id 371d9cc10c6035af6eb27395ff12d8a58978e6a776f179a4c5b48a5f75b0e62c
Jul 31 14:12:44.511127 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:44.510802    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d76362e_39cc_4a7a_88df_8902bf386dc1.slice/crio-5585bf201f12f42e49bd236c2b8adaa88f1fcaff7c4ce19fef8f4f814eddc617 WatchSource:0}: Error finding container 5585bf201f12f42e49bd236c2b8adaa88f1fcaff7c4ce19fef8f4f814eddc617: Status 404 returned error can't find the container with id 5585bf201f12f42e49bd236c2b8adaa88f1fcaff7c4ce19fef8f4f814eddc617
Jul 31 14:12:44.512606 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:44.512576    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ea6151_ad56_41c1_8ed5_ede42ab94632.slice/crio-442e62a7bd6abac50c3b1235a2a2c119e964659188a0e8d005f2f339e4ab90e5 WatchSource:0}: Error finding container 442e62a7bd6abac50c3b1235a2a2c119e964659188a0e8d005f2f339e4ab90e5: Status 404 returned error can't find the container with id 442e62a7bd6abac50c3b1235a2a2c119e964659188a0e8d005f2f339e4ab90e5
Jul 31 14:12:44.513784 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:12:44.513738    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8713ceba_51f1_4cf6_8ea4_4768ebce9cd4.slice/crio-ee7d03adc73117b9c67351fcd830af927d6020e326b15bcdd3547cc55b6863ce WatchSource:0}: Error finding container ee7d03adc73117b9c67351fcd830af927d6020e326b15bcdd3547cc55b6863ce: Status 404 returned error can't find the container with id ee7d03adc73117b9c67351fcd830af927d6020e326b15bcdd3547cc55b6863ce
Jul 31 14:12:44.573676 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.573657    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:44.573773 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:44.573754    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:44.581211 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.581181    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x" event={"ID":"2ba92576-73a7-4597-964d-d18042bd4228","Type":"ContainerStarted","Data":"0dd54933375effbedbf6cb9764765342a5d2d7a9de1e35b8732cca89292bc34b"}
Jul 31 14:12:44.582724 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.582701    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hlrmr" event={"ID":"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4","Type":"ContainerStarted","Data":"ee7d03adc73117b9c67351fcd830af927d6020e326b15bcdd3547cc55b6863ce"}
Jul 31 14:12:44.584514 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.584472    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" event={"ID":"37ea6151-ad56-41c1-8ed5-ede42ab94632","Type":"ContainerStarted","Data":"442e62a7bd6abac50c3b1235a2a2c119e964659188a0e8d005f2f339e4ab90e5"}
Jul 31 14:12:44.586232 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.586208    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zlw8q" event={"ID":"3d76362e-39cc-4a7a-88df-8902bf386dc1","Type":"ContainerStarted","Data":"5585bf201f12f42e49bd236c2b8adaa88f1fcaff7c4ce19fef8f4f814eddc617"}
Jul 31 14:12:44.587539 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.587516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rtc4t" event={"ID":"57c7c983-8b4d-4494-be24-c63071b5ad0a","Type":"ContainerStarted","Data":"371d9cc10c6035af6eb27395ff12d8a58978e6a776f179a4c5b48a5f75b0e62c"}
Jul 31 14:12:44.588746 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.588719    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"7d1cdfb5c91d0612a62a9a5048df375fd53d95c4a794e9a330076be60e1b9bd8"}
Jul 31 14:12:44.589865 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.589842    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gkw4n" event={"ID":"66f05e4e-0197-4037-923c-c626b7982ab2","Type":"ContainerStarted","Data":"45da7f53e5738a5e591178d2accc5cc1a3dafddb0bba3cbbbeb2fc4901590885"}
Jul 31 14:12:44.590955 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.590931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wccvh" event={"ID":"83810d6e-f7ae-46d0-a38c-4bab260b035d","Type":"ContainerStarted","Data":"0d4c6ee59dc14367a6d68c6429b68fa69900af3b01211fe4b54ab53322712841"}
Jul 31 14:12:44.592235 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:44.592211    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tx8nv" event={"ID":"84db4ce3-b047-40a8-a504-1bd5651ffba0","Type":"ContainerStarted","Data":"d4aedf707a14cabda2da32e5ff3486891e57c1098b9e040a331fb7419e7ea42f"}
Jul 31 14:12:45.087937 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.087615    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:45.087937 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.087895    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:12:45.087937 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.087921    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:12:45.087937 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.087935    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pjj9m for pod openshift-network-diagnostics/network-check-target-xxbnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:45.088280 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.087958    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:45.088280 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.087995    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m podName:6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f nodeName:}" failed. No retries permitted until 2026-07-31 14:12:47.087975797 +0000 UTC m=+5.960539043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pjj9m" (UniqueName: "kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m") pod "network-check-target-xxbnh" (UID: "6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:45.088280 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.088093    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:45.088280 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.088142    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret podName:1a778adb-72c6-427b-97e3-9da9685b0467 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:47.088126583 +0000 UTC m=+5.960689829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret") pod "global-pull-secret-syncer-sfw5j" (UID: "1a778adb-72c6-427b-97e3-9da9685b0467") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:45.188545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.188454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:45.188732 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.188613    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:45.188732 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.188675    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:12:47.188657648 +0000 UTC m=+6.061220886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:45.507081 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.506952    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:07:43 +0000 UTC" deadline="2027-12-30 10:25:51.482264353 +0000 UTC"
Jul 31 14:12:45.507081 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.507033    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12404h13m5.975250278s"
Jul 31 14:12:45.576704 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.576085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:45.576704 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.576210    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:45.576704 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.576575    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:45.576704 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:45.576659    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:12:45.612011 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.611690    2558 generic.go:356] "Generic (PLEG): container finished" podID="d443483119bcac6f2b2616712ae7bca3" containerID="536143a239a4d8fc741189bc71a995e5501ea2ac7527d2c41773b8c2c8e0468c" exitCode=0
Jul 31 14:12:45.612011 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.611787    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal" event={"ID":"d443483119bcac6f2b2616712ae7bca3","Type":"ContainerDied","Data":"536143a239a4d8fc741189bc71a995e5501ea2ac7527d2c41773b8c2c8e0468c"}
Jul 31 14:12:45.622793 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.622657    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal" event={"ID":"0fcb5a47836661934246ad400500ba6e","Type":"ContainerStarted","Data":"7a4f709543dc4f7f22e07543f61cfe92a80c26b1740bfd6d13d62177a9235a03"}
Jul 31 14:12:45.641712 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:45.641650    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-143.ec2.internal" podStartSLOduration=3.641629363 podStartE2EDuration="3.641629363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:45.640936528 +0000 UTC m=+4.513499774" watchObservedRunningTime="2026-07-31 14:12:45.641629363 +0000 UTC m=+4.514192615"
Jul 31 14:12:46.574403 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:46.573885    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:46.574403 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:46.574041    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:46.653804 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:46.653769    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal" event={"ID":"d443483119bcac6f2b2616712ae7bca3","Type":"ContainerStarted","Data":"904e7144e03677079435f905edba953b63348afdc762270e7caf151bafcf471b"}
Jul 31 14:12:47.105663 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:47.105626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:47.105832 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:47.105694    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:47.105915 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.105880    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:12:47.105915 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.105899    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:12:47.105915 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.105912    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pjj9m for pod openshift-network-diagnostics/network-check-target-xxbnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:47.106086 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.105971    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m podName:6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f nodeName:}" failed. No retries permitted until 2026-07-31 14:12:51.105951529 +0000 UTC m=+9.978514767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pjj9m" (UniqueName: "kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m") pod "network-check-target-xxbnh" (UID: "6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:47.106398 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.106379    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:47.106480 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.106434    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret podName:1a778adb-72c6-427b-97e3-9da9685b0467 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:51.106417326 +0000 UTC m=+9.978980562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret") pod "global-pull-secret-syncer-sfw5j" (UID: "1a778adb-72c6-427b-97e3-9da9685b0467") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:47.206982 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:47.206373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:47.206982 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.206574    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:47.206982 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.206640    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:12:51.206622452 +0000 UTC m=+10.079185686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:47.574229 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:47.574153    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:47.574229 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:47.574196    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:47.574443 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.574306    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:47.574443 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:47.574425    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:12:48.573646 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:48.573607    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:48.573818 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:48.573773    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:49.573157 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:49.573123    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:49.573617 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:49.573272    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:49.573617 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:49.573316    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:49.573617 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:49.573430    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:12:50.573495 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:50.573403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:50.573960 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:50.573547    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:51.143336 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:51.142987    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:51.143336 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:51.143140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:51.143336 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.143162    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:12:51.143336 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.143190    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:12:51.143336 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.143203    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pjj9m for pod openshift-network-diagnostics/network-check-target-xxbnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:51.143336 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.143248    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:51.143336 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.143266    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m podName:6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f nodeName:}" failed. No retries permitted until 2026-07-31 14:12:59.143244279 +0000 UTC m=+18.015807736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pjj9m" (UniqueName: "kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m") pod "network-check-target-xxbnh" (UID: "6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:51.143336 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.143298    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret podName:1a778adb-72c6-427b-97e3-9da9685b0467 nodeName:}" failed. No retries permitted until 2026-07-31 14:12:59.14328172 +0000 UTC m=+18.015844956 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret") pod "global-pull-secret-syncer-sfw5j" (UID: "1a778adb-72c6-427b-97e3-9da9685b0467") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:51.243704 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:51.243669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:51.243890 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.243866    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:51.243957 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.243925    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:12:59.243906725 +0000 UTC m=+18.116469962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:51.574942 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:51.574235    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:51.574942 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.574355    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:12:51.574942 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:51.574740    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:51.574942 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:51.574846    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:52.573988 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:52.573472    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:52.573988 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:52.573616    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:53.573371 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:53.573340    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:53.573829 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:53.573466    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:53.573829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:53.573730    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:53.573948 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:53.573864    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:12:54.573831 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:54.573791    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:54.574251 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:54.573954    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:55.573250 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:55.573217    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:55.573433 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:55.573342    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:55.573433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:55.573421    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:55.573543 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:55.573518    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:12:56.573058 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:56.573015    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:56.573578 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:56.573166    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:57.572871 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:57.572835    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:57.573055 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:57.572959    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:57.573055 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:57.573032    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:57.573450 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:57.573118    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:12:58.573076 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:58.573046    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:58.573305 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:58.573149    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:12:59.206256 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:59.206216    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:59.206467 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:59.206277    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:59.206467 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.206391    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:59.206467 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.206462    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret podName:1a778adb-72c6-427b-97e3-9da9685b0467 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:15.206441864 +0000 UTC m=+34.079005118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret") pod "global-pull-secret-syncer-sfw5j" (UID: "1a778adb-72c6-427b-97e3-9da9685b0467") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:12:59.206620 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.206402    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:12:59.206620 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.206501    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:12:59.206620 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.206516    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pjj9m for pod openshift-network-diagnostics/network-check-target-xxbnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:59.206620 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.206572    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m podName:6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f nodeName:}" failed. No retries permitted until 2026-07-31 14:13:15.206557029 +0000 UTC m=+34.079120267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pjj9m" (UniqueName: "kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m") pod "network-check-target-xxbnh" (UID: "6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:12:59.307462 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:59.307425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:12:59.307641 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.307580    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:59.307697 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.307658    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:13:15.307635921 +0000 UTC m=+34.180199152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:12:59.573304 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:59.573229    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:12:59.573304 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:12:59.573270    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:12:59.573774 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.573347    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:12:59.573774 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:12:59.573481    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:00.573209 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:00.573169    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:00.573395 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:00.573320    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:01.574053 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:01.573946    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:01.574519 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:01.574078    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:13:01.574519 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:01.574135    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:01.574519 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:01.574251    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:02.573561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:02.573523    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:02.573749 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:02.573666    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:03.573526 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.573198    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:03.574198 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.573260    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:03.574198 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:03.573608    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:13:03.574198 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:03.573674    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:03.688539 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.688438    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tx8nv" event={"ID":"84db4ce3-b047-40a8-a504-1bd5651ffba0","Type":"ContainerStarted","Data":"3951749fcb2b870fa5ff46df4ba78bb5951fd410437907670bc115eedddb0cbe"}
Jul 31 14:13:03.690313 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.690283    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x" event={"ID":"2ba92576-73a7-4597-964d-d18042bd4228","Type":"ContainerStarted","Data":"6092fc7782ae5968d624689cdb913a922a223c66f9a7bb9ab4c62cc203175ac3"}
Jul 31 14:13:03.691914 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.691889    2558 generic.go:356] "Generic (PLEG): container finished" podID="37ea6151-ad56-41c1-8ed5-ede42ab94632" containerID="23b59104b67fcae3770b7da2103fe6294472ee01e37a4861bb98d7af8255443c" exitCode=0
Jul 31 14:13:03.692041 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.691974    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" event={"ID":"37ea6151-ad56-41c1-8ed5-ede42ab94632","Type":"ContainerDied","Data":"23b59104b67fcae3770b7da2103fe6294472ee01e37a4861bb98d7af8255443c"}
Jul 31 14:13:03.693702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.693637    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zlw8q" event={"ID":"3d76362e-39cc-4a7a-88df-8902bf386dc1","Type":"ContainerStarted","Data":"ad209143171ad7d06103f7e428410d0ea158b6552dc7c1622f31209f2e2af2b8"}
Jul 31 14:13:03.695313 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.695290    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rtc4t" event={"ID":"57c7c983-8b4d-4494-be24-c63071b5ad0a","Type":"ContainerStarted","Data":"64cdbd894f9657a84bdc5fd60b45b3c6065344bdd605b30110fc6d9156946bcb"}
Jul 31 14:13:03.698723 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.698697    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"4f7025e686062c8c4deb6090155c0f246d367155afe2e4e031222b5d7f5042ad"}
Jul 31 14:13:03.698828 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.698768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"2652a986f91f9670895b3b9a15fb164ec7e49aca6f75f83d485b3bdc5d0a241a"}
Jul 31 14:13:03.698828 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.698819    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"c8b81bfed12f87e221a2b2f329fb6d80a2ad9a552ab25a7b3981dfc55188d8d9"}
Jul 31 14:13:03.698943 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.698867    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"467849f98707c00eaa82acfcd7e9d390f272a9cde0175f419c1546f473682bce"}
Jul 31 14:13:03.698943 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.698911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"443f94299296132d2a7ef728bf1a25239aebb4db820a3aebf67f44b4caf56563"}
Jul 31 14:13:03.699060 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.698958    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"e50e75679c9761eb010a8c89598093d690bdecd526845d5a29569e226d6ba762"}
Jul 31 14:13:03.700286 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.700246    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gkw4n" event={"ID":"66f05e4e-0197-4037-923c-c626b7982ab2","Type":"ContainerStarted","Data":"d580d135e4aabf584a521a269c5c277459bd0e13dc4a81c2eec370ec0a89c8f5"}
Jul 31 14:13:03.703297 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.703273    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wccvh" event={"ID":"83810d6e-f7ae-46d0-a38c-4bab260b035d","Type":"ContainerStarted","Data":"7a4f4fe1b4660e0b071fd840c88d69dc7710a5f721bc5dfd5ee4c3f07763de46"}
Jul 31 14:13:03.708607 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.708563    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tx8nv" podStartSLOduration=4.435902559 podStartE2EDuration="22.70855164s" totalImagesPullingTime="18.272649081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:03.708503411 +0000 UTC m=+22.581066665" watchObservedRunningTime="2026-07-31 14:13:03.70855164 +0000 UTC m=+22.581114892"
Jul 31 14:13:03.708959 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.708935    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-143.ec2.internal" podStartSLOduration=21.708925292 podStartE2EDuration="21.708925292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:12:46.668675363 +0000 UTC m=+5.541238617" watchObservedRunningTime="2026-07-31 14:13:03.708925292 +0000 UTC m=+22.581488543"
Jul 31 14:13:03.728163 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.728119    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wccvh" podStartSLOduration=4.406375226 podStartE2EDuration="22.728105387s" totalImagesPullingTime="18.321730161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:03.72778944 +0000 UTC m=+22.600352694" watchObservedRunningTime="2026-07-31 14:13:03.728105387 +0000 UTC m=+22.600668639"
Jul 31 14:13:03.742506 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.742455    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zlw8q" podStartSLOduration=8.650963747 podStartE2EDuration="22.742442687s" totalImagesPullingTime="14.09147894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:03.74208501 +0000 UTC m=+22.614648263" watchObservedRunningTime="2026-07-31 14:13:03.742442687 +0000 UTC m=+22.615005940"
Jul 31 14:13:03.781036 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.780984    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gkw4n" podStartSLOduration=4.511257963 podStartE2EDuration="22.780968965s" totalImagesPullingTime="18.269711002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:03.780562353 +0000 UTC m=+22.653125614" watchObservedRunningTime="2026-07-31 14:13:03.780968965 +0000 UTC m=+22.653532218"
Jul 31 14:13:03.796311 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:03.796266    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rtc4t" podStartSLOduration=4.530580024 podStartE2EDuration="22.796252334s" totalImagesPullingTime="18.26567231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:03.795595686 +0000 UTC m=+22.668158930" watchObservedRunningTime="2026-07-31 14:13:03.796252334 +0000 UTC m=+22.668815578"
Jul 31 14:13:04.213209 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:04.213179    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 14:13:04.543445 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:04.543292    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:13:04.213197326Z","UUID":"1970e210-fdf1-4f95-894b-6007d99d099e","Handler":null,"Name":"","Endpoint":""}
Jul 31 14:13:04.545041 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:04.545003    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 14:13:04.545164 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:04.545067    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 14:13:04.573565 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:04.573541    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:04.573979 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:04.573649    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:04.707885 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:04.707850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x" event={"ID":"2ba92576-73a7-4597-964d-d18042bd4228","Type":"ContainerStarted","Data":"5b466af3a07450cabbf91632fb158ca1e1778c99cfad346088a574bbb415631f"}
Jul 31 14:13:04.709801 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:04.709770    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hlrmr" event={"ID":"8713ceba-51f1-4cf6-8ea4-4768ebce9cd4","Type":"ContainerStarted","Data":"5526f4b41b7b0ffc5fec1fbd485615c49b1ed30d97ef3f47e6214639717eecdb"}
Jul 31 14:13:04.725769 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:04.725716    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hlrmr" podStartSLOduration=5.464919609 podStartE2EDuration="23.725698164s" totalImagesPullingTime="18.260778555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:04.725468137 +0000 UTC m=+23.598031391" watchObservedRunningTime="2026-07-31 14:13:04.725698164 +0000 UTC m=+23.598261420"
Jul 31 14:13:05.573892 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:05.573489    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:05.573892 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:05.573529    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:05.573892 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:05.573611    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:13:05.573892 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:05.573744    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:05.715012 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:05.714976    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x" event={"ID":"2ba92576-73a7-4597-964d-d18042bd4228","Type":"ContainerStarted","Data":"ca16d4e592d90142760d6b9c3c006540933709f3bdf86cf4a501a07ccdc099a5"}
Jul 31 14:13:05.719041 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:05.719003    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"db0ac12d5dc57c1bb771e8d97ae3f3795548145d5fc2d1b4d25524a7aee71b2d"}
Jul 31 14:13:05.732589 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:05.732546    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6j55x" podStartSLOduration=4.18320399 podStartE2EDuration="24.732528383s" totalImagesPullingTime="20.549324393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:05.731934583 +0000 UTC m=+24.604497848" watchObservedRunningTime="2026-07-31 14:13:05.732528383 +0000 UTC m=+24.605091635"
Jul 31 14:13:06.515956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:06.515923    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:13:06.516471 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:06.516452    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:13:06.573288 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:06.573044    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:06.573445 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:06.573401    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:06.721188 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:06.721153    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:13:06.721578 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:06.721481    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zlw8q"
Jul 31 14:13:07.572979 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:07.572943    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:07.573155 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:07.573119    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:13:07.573155 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:07.573136    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:07.573271 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:07.573249    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:08.573067 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:08.573042    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:08.573419 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:08.573139    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:08.728409 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:08.728375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" event={"ID":"bb0c96e2-5e54-454f-b865-a7ee847852d2","Type":"ContainerStarted","Data":"3f3416353f9e005f6a32f59cc8e0ffcbb979439ee5f41855a25e8181afc889ef"}
Jul 31 14:13:08.757742 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:08.757680    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj" podStartSLOduration=9.428544582 podStartE2EDuration="27.757657972s" totalImagesPullingTime="18.32911339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:08.757394326 +0000 UTC m=+27.629957579" watchObservedRunningTime="2026-07-31 14:13:08.757657972 +0000 UTC m=+27.630221225"
Jul 31 14:13:09.573450 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.573273    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:09.574038 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.573338    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:09.574038 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:09.573529    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:13:09.574038 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:09.573587    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:09.732620 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.732585    2558 generic.go:356] "Generic (PLEG): container finished" podID="37ea6151-ad56-41c1-8ed5-ede42ab94632" containerID="589f0d27b994857a47d4f746968b8775ba44d676c28051b3a0c850e4766e9c0e" exitCode=0
Jul 31 14:13:09.732778 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.732672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" event={"ID":"37ea6151-ad56-41c1-8ed5-ede42ab94632","Type":"ContainerDied","Data":"589f0d27b994857a47d4f746968b8775ba44d676c28051b3a0c850e4766e9c0e"}
Jul 31 14:13:09.733381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.733222    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:13:09.733381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.733307    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:13:09.733381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.733357    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:13:09.749217 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.749189    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:13:09.749351 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:09.749300    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:13:10.573159 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:10.573125    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:10.573355 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:10.573253    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:10.608720 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:10.608675    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jbbns"]
Jul 31 14:13:10.609537 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:10.609510    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sfw5j"]
Jul 31 14:13:10.609674 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:10.609615    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:10.609758 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:10.609720    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:13:10.610636 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:10.610609    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xxbnh"]
Jul 31 14:13:10.610738 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:10.610696    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:10.610866 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:10.610793    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:10.734576 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:10.734497    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:10.734733 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:10.734633    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:11.738794 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:11.738758    2558 generic.go:356] "Generic (PLEG): container finished" podID="37ea6151-ad56-41c1-8ed5-ede42ab94632" containerID="ece547e5b0badc00f7e24d1d6b060c3683bca86567b71e121ef48b36193cf8eb" exitCode=0
Jul 31 14:13:11.739178 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:11.738850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" event={"ID":"37ea6151-ad56-41c1-8ed5-ede42ab94632","Type":"ContainerDied","Data":"ece547e5b0badc00f7e24d1d6b060c3683bca86567b71e121ef48b36193cf8eb"}
Jul 31 14:13:12.573818 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:12.573736    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:12.573992 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:12.573736    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:12.573992 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:12.573869    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:12.573992 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:12.573914    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:12.573992 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:12.573740    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:12.574181 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:12.574047    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:13:13.746136 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:13.745879    2558 generic.go:356] "Generic (PLEG): container finished" podID="37ea6151-ad56-41c1-8ed5-ede42ab94632" containerID="c43d63a973c76f21e6c8a1f782bd87068bafc3a835e61da53b1b3273e7b1e1e0" exitCode=0
Jul 31 14:13:13.746136 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:13.745945    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" event={"ID":"37ea6151-ad56-41c1-8ed5-ede42ab94632","Type":"ContainerDied","Data":"c43d63a973c76f21e6c8a1f782bd87068bafc3a835e61da53b1b3273e7b1e1e0"}
Jul 31 14:13:14.573676 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:14.573642    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:14.573853 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:14.573641    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:14.573853 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:14.573794    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:13:14.573853 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:14.573641    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:14.574035 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:14.573828    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sfw5j" podUID="1a778adb-72c6-427b-97e3-9da9685b0467"
Jul 31 14:13:14.574035 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:14.573917    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxbnh" podUID="6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f"
Jul 31 14:13:15.230431 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.230398    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:15.230824 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.230460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:15.230824 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.230590    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:13:15.230824 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.230652    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret podName:1a778adb-72c6-427b-97e3-9da9685b0467 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:47.230632674 +0000 UTC m=+66.103195912 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret") pod "global-pull-secret-syncer-sfw5j" (UID: "1a778adb-72c6-427b-97e3-9da9685b0467") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:13:15.230824 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.230741    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:13:15.230824 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.230753    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:13:15.230824 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.230761    2558 projected.go:196] Error preparing data for projected volume kube-api-access-pjj9m for pod openshift-network-diagnostics/network-check-target-xxbnh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:13:15.230824 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.230800    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m podName:6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f nodeName:}" failed. No retries permitted until 2026-07-31 14:13:47.230787456 +0000 UTC m=+66.103350686 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pjj9m" (UniqueName: "kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m") pod "network-check-target-xxbnh" (UID: "6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:13:15.331364 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.331338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:15.331514 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.331495    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:13:15.331573 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.331568    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:13:47.331547193 +0000 UTC m=+66.204110432 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:13:15.518041 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.517950    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-143.ec2.internal" event="NodeReady"
Jul 31 14:13:15.518209 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.518114    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 14:13:15.558840 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.558810    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fmdw8"]
Jul 31 14:13:15.560888 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.560849    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.561497 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.561472    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m98v2"]
Jul 31 14:13:15.563127 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.563109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m98v2"
Jul 31 14:13:15.563347 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.563275    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 14:13:15.563886 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.563867    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9vdf\""
Jul 31 14:13:15.564210 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.564191    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 14:13:15.565662 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.565639    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p49zl\""
Jul 31 14:13:15.565760 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.565742    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 14:13:15.565877 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.565860    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 14:13:15.566086 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.566065    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 14:13:15.571759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.571666    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fmdw8"]
Jul 31 14:13:15.578104 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.578085    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m98v2"]
Jul 31 14:13:15.733924 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.733883    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-config-volume\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.734125 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.733944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4m6\" (UniqueName: \"kubernetes.io/projected/96a9dc2b-3f07-4be6-85a0-722f0b87c5ac-kube-api-access-ff4m6\") pod \"ingress-canary-m98v2\" (UID: \"96a9dc2b-3f07-4be6-85a0-722f0b87c5ac\") " pod="openshift-ingress-canary/ingress-canary-m98v2"
Jul 31 14:13:15.734125 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.733984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96a9dc2b-3f07-4be6-85a0-722f0b87c5ac-cert\") pod \"ingress-canary-m98v2\" (UID: \"96a9dc2b-3f07-4be6-85a0-722f0b87c5ac\") " pod="openshift-ingress-canary/ingress-canary-m98v2"
Jul 31 14:13:15.734125 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.734010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.734125 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.734106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2v9\" (UniqueName: \"kubernetes.io/projected/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-kube-api-access-hj2v9\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.734334 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.734170    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-tmp-dir\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.835348 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.835315    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-tmp-dir\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.835519 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.835370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-config-volume\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.835519 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.835393    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4m6\" (UniqueName: \"kubernetes.io/projected/96a9dc2b-3f07-4be6-85a0-722f0b87c5ac-kube-api-access-ff4m6\") pod \"ingress-canary-m98v2\" (UID: \"96a9dc2b-3f07-4be6-85a0-722f0b87c5ac\") " pod="openshift-ingress-canary/ingress-canary-m98v2"
Jul 31 14:13:15.835519 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.835416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96a9dc2b-3f07-4be6-85a0-722f0b87c5ac-cert\") pod \"ingress-canary-m98v2\" (UID: \"96a9dc2b-3f07-4be6-85a0-722f0b87c5ac\") " pod="openshift-ingress-canary/ingress-canary-m98v2"
Jul 31 14:13:15.835519 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.835438    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.835519 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.835468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2v9\" (UniqueName: \"kubernetes.io/projected/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-kube-api-access-hj2v9\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.835787 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.835612    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:13:15.835787 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:15.835698    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:16.335676656 +0000 UTC m=+35.208239888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:13:15.835787 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.835701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-tmp-dir\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.835996 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.835973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-config-volume\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.839745 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.839720    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96a9dc2b-3f07-4be6-85a0-722f0b87c5ac-cert\") pod \"ingress-canary-m98v2\" (UID: \"96a9dc2b-3f07-4be6-85a0-722f0b87c5ac\") " pod="openshift-ingress-canary/ingress-canary-m98v2"
Jul 31 14:13:15.844293 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.844262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4m6\" (UniqueName: \"kubernetes.io/projected/96a9dc2b-3f07-4be6-85a0-722f0b87c5ac-kube-api-access-ff4m6\") pod \"ingress-canary-m98v2\" (UID: \"96a9dc2b-3f07-4be6-85a0-722f0b87c5ac\") " pod="openshift-ingress-canary/ingress-canary-m98v2"
Jul 31 14:13:15.844491 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.844468    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2v9\" (UniqueName: \"kubernetes.io/projected/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-kube-api-access-hj2v9\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:15.881654 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:15.881621    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m98v2"
Jul 31 14:13:16.030530 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.030492    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m98v2"]
Jul 31 14:13:16.034408 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:13:16.034380    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a9dc2b_3f07_4be6_85a0_722f0b87c5ac.slice/crio-c6996eb7690cf95f66b3c0094de56f66a1381437968b61739ff3012a8b619793 WatchSource:0}: Error finding container c6996eb7690cf95f66b3c0094de56f66a1381437968b61739ff3012a8b619793: Status 404 returned error can't find the container with id c6996eb7690cf95f66b3c0094de56f66a1381437968b61739ff3012a8b619793
Jul 31 14:13:16.339068 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.339016    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:16.339874 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:16.339183    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:13:16.339874 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:16.339256    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:17.339238596 +0000 UTC m=+36.211801847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:13:16.573602 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.573566    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:16.573602 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.573600    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:16.573803 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.573566    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:16.576255 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.576234    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:13:16.577588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.577372    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lwbh\""
Jul 31 14:13:16.577588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.577383    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:13:16.577588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.577439    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:13:16.577588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.577471    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:13:16.577588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.577522    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rdxfb\""
Jul 31 14:13:16.754493 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:16.754458    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m98v2" event={"ID":"96a9dc2b-3f07-4be6-85a0-722f0b87c5ac","Type":"ContainerStarted","Data":"c6996eb7690cf95f66b3c0094de56f66a1381437968b61739ff3012a8b619793"}
Jul 31 14:13:17.347579 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:17.347541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:17.348049 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:17.347698    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:13:17.348049 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:17.347758    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:19.347744643 +0000 UTC m=+38.220307876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:13:18.761814 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:18.761778    2558 generic.go:356] "Generic (PLEG): container finished" podID="37ea6151-ad56-41c1-8ed5-ede42ab94632" containerID="4d7bba9c312fdfd4dc06c37895c8ca04c12f67512f8592cc222002eef2c86c07" exitCode=0
Jul 31 14:13:18.762193 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:18.761834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" event={"ID":"37ea6151-ad56-41c1-8ed5-ede42ab94632","Type":"ContainerDied","Data":"4d7bba9c312fdfd4dc06c37895c8ca04c12f67512f8592cc222002eef2c86c07"}
Jul 31 14:13:19.361728 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:19.361690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:19.361963 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:19.361868    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:13:19.361963 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:19.361948    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:23.361926457 +0000 UTC m=+42.234489694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:13:19.771494 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:19.771408    2558 generic.go:356] "Generic (PLEG): container finished" podID="37ea6151-ad56-41c1-8ed5-ede42ab94632" containerID="5e04c722f6688a72c937a1c8eac4d81d896c48ce34595f5429656f5a52b541b4" exitCode=0
Jul 31 14:13:19.771494 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:19.771457    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" event={"ID":"37ea6151-ad56-41c1-8ed5-ede42ab94632","Type":"ContainerDied","Data":"5e04c722f6688a72c937a1c8eac4d81d896c48ce34595f5429656f5a52b541b4"}
Jul 31 14:13:20.776836 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:20.776798    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" event={"ID":"37ea6151-ad56-41c1-8ed5-ede42ab94632","Type":"ContainerStarted","Data":"cb486c699eddad93518456b34ae135804f7eca31319b4f220196769a6bde35b3"}
Jul 31 14:13:20.778172 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:20.778152    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m98v2" event={"ID":"96a9dc2b-3f07-4be6-85a0-722f0b87c5ac","Type":"ContainerStarted","Data":"46edf91027d72d009a80ae7c8c2fa4bea678a0138b9bb9fad79b1b9c22437da6"}
Jul 31 14:13:20.799187 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:20.799141    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wp2xh" podStartSLOduration=9.338003413 podStartE2EDuration="39.799122389s" totalImagesPullingTime="30.461118976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:20.797356033 +0000 UTC m=+39.669919286" watchObservedRunningTime="2026-07-31 14:13:20.799122389 +0000 UTC m=+39.671685643"
Jul 31 14:13:20.814780 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:20.814734    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m98v2" podStartSLOduration=1.945878338 podStartE2EDuration="5.814720172s" totalImagesPullingTime="3.868841834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:20.813982214 +0000 UTC m=+39.686545467" watchObservedRunningTime="2026-07-31 14:13:20.814720172 +0000 UTC m=+39.687283455"
Jul 31 14:13:23.390408 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:23.390371    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:23.390794 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:23.390500    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:13:23.390794 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:23.390551    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:31.390537129 +0000 UTC m=+50.263100359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:13:31.442897 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:31.442852    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:31.443382 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:31.442979    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:13:31.443382 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:31.443047    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:13:47.443018028 +0000 UTC m=+66.315581259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:13:41.758977 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:41.758952    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zvkmj"
Jul 31 14:13:47.249407 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.249357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:47.249847 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.249438    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:47.252334 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.252314    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:13:47.252443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.252429    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:13:47.262528 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.262503    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:13:47.263892 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.263870    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1a778adb-72c6-427b-97e3-9da9685b0467-original-pull-secret\") pod \"global-pull-secret-syncer-sfw5j\" (UID: \"1a778adb-72c6-427b-97e3-9da9685b0467\") " pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:47.273504 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.273477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjj9m\" (UniqueName: \"kubernetes.io/projected/6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f-kube-api-access-pjj9m\") pod \"network-check-target-xxbnh\" (UID: \"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f\") " pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:47.350234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.350190    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:13:47.353120 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.353105    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:13:47.361310 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:47.361289    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 14:13:47.361388 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:47.361364    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:14:51.361344774 +0000 UTC m=+130.233908007 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : secret "metrics-daemon-secret" not found
Jul 31 14:13:47.450899 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.450853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:13:47.451044 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:47.450995    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:13:47.451088 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:13:47.451059    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:14:19.451046684 +0000 UTC m=+98.323609915 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:13:47.486177 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.486152    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sfw5j"
Jul 31 14:13:47.500204 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.500186    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rdxfb\""
Jul 31 14:13:47.508315 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.508296    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:47.620286 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.620254    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sfw5j"]
Jul 31 14:13:47.623145 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:13:47.623093    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a778adb_72c6_427b_97e3_9da9685b0467.slice/crio-45b094fe37ac444ca71c04023a6b01629ab54af718ce805b567261065be06165 WatchSource:0}: Error finding container 45b094fe37ac444ca71c04023a6b01629ab54af718ce805b567261065be06165: Status 404 returned error can't find the container with id 45b094fe37ac444ca71c04023a6b01629ab54af718ce805b567261065be06165
Jul 31 14:13:47.635162 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.635140    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xxbnh"]
Jul 31 14:13:47.637853 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:13:47.637828    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6407d04f_46ee_4f64_ac0d_18f8a8fa3e8f.slice/crio-474afcf31fccec8f096f19f7f72583e18ea1bd24b3458be823c1c03cc8c20730 WatchSource:0}: Error finding container 474afcf31fccec8f096f19f7f72583e18ea1bd24b3458be823c1c03cc8c20730: Status 404 returned error can't find the container with id 474afcf31fccec8f096f19f7f72583e18ea1bd24b3458be823c1c03cc8c20730
Jul 31 14:13:47.838573 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.838533    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xxbnh" event={"ID":"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f","Type":"ContainerStarted","Data":"474afcf31fccec8f096f19f7f72583e18ea1bd24b3458be823c1c03cc8c20730"}
Jul 31 14:13:47.839529 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:47.839504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sfw5j" event={"ID":"1a778adb-72c6-427b-97e3-9da9685b0467","Type":"ContainerStarted","Data":"45b094fe37ac444ca71c04023a6b01629ab54af718ce805b567261065be06165"}
Jul 31 14:13:52.854490 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:52.854447    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xxbnh" event={"ID":"6407d04f-46ee-4f64-ac0d-18f8a8fa3e8f","Type":"ContainerStarted","Data":"13655e867b1586e6849757868a105fe0614722e530f70e5ab3a90ceba2737dba"}
Jul 31 14:13:52.854956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:52.854549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:13:52.855880 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:52.855856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sfw5j" event={"ID":"1a778adb-72c6-427b-97e3-9da9685b0467","Type":"ContainerStarted","Data":"876a4311b8f9b53bff9d44a21cf547d078f3daa9afdaec17bfab5ac5dc037522"}
Jul 31 14:13:52.869770 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:52.869729    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xxbnh" podStartSLOduration=67.33998011 podStartE2EDuration="1m11.869717295s" totalImagesPullingTime="4.529737185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:52.869285959 +0000 UTC m=+71.741849212" watchObservedRunningTime="2026-07-31 14:13:52.869717295 +0000 UTC m=+71.742280547"
Jul 31 14:13:52.883303 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:13:52.883254    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sfw5j" podStartSLOduration=67.335675233 podStartE2EDuration="1m11.883242934s" totalImagesPullingTime="4.547567701s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:13:52.882628731 +0000 UTC m=+71.755192005" watchObservedRunningTime="2026-07-31 14:13:52.883242934 +0000 UTC m=+71.755806186"
Jul 31 14:14:19.467161 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:14:19.467126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:14:19.467640 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:14:19.467232    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:14:19.467640 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:14:19.467297    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:15:23.46728359 +0000 UTC m=+162.339846821 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:14:23.861382 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:14:23.861354    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xxbnh"
Jul 31 14:14:51.380406 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:14:51.380367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:14:51.380890 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:14:51.380511    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 14:14:51.380890 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:14:51.380592    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs podName:13d90876-b172-48d8-95e3-420e703c71cf nodeName:}" failed. No retries permitted until 2026-07-31 14:16:53.380575554 +0000 UTC m=+252.253138785 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs") pod "network-metrics-daemon-jbbns" (UID: "13d90876-b172-48d8-95e3-420e703c71cf") : secret "metrics-daemon-secret" not found
Jul 31 14:15:18.573713 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:15:18.573653    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fmdw8" podUID="efbfc503-7eb7-435f-94a8-49b4fc9ecdd4"
Jul 31 14:15:19.056926 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:15:19.056895    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:15:19.592517 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:15:19.592467    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jbbns" podUID="13d90876-b172-48d8-95e3-420e703c71cf"
Jul 31 14:15:23.496877 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:15:23.496826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:15:23.497431 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:15:23.496968    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:15:23.497431 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:15:23.497069    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls podName:efbfc503-7eb7-435f-94a8-49b4fc9ecdd4 nodeName:}" failed. No retries permitted until 2026-07-31 14:17:25.49704524 +0000 UTC m=+284.369608489 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls") pod "dns-default-fmdw8" (UID: "efbfc503-7eb7-435f-94a8-49b4fc9ecdd4") : secret "dns-default-metrics-tls" not found
Jul 31 14:15:32.573614 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:15:32.573579    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:16:29.767382 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.767351    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-ll4lr"]
Jul 31 14:16:29.769271 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.769252    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.771966 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.771940    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x9dlt\""
Jul 31 14:16:29.772079 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.771994    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:16:29.772079 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.772001    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 31 14:16:29.772079 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.771999    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 31 14:16:29.774146 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.774117    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 31 14:16:29.776514 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.776496    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"]
Jul 31 14:16:29.782121 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.782102    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 31 14:16:29.782216 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.782125    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"]
Jul 31 14:16:29.782283 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.782245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.783785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.783771    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"]
Jul 31 14:16:29.783907 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.783886    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:29.785896 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.785867    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:29.786223 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.786205    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-ll4lr"]
Jul 31 14:16:29.786519 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.786503    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 31 14:16:29.786717 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.786704    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 14:16:29.786768 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.786734    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-czzp4\""
Jul 31 14:16:29.786768 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.786741    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nlvv2\""
Jul 31 14:16:29.787064 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.787044    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 31 14:16:29.787176 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.787091    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 31 14:16:29.787405 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.787385    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:16:29.789124 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.788792    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 31 14:16:29.789124 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.788901    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 14:16:29.789124 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.788903    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 31 14:16:29.789124 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.788976    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 31 14:16:29.789423 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.789240    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 31 14:16:29.789478 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.789459    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-26w8d\""
Jul 31 14:16:29.789944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.789770    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:16:29.790356 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.790336    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"]
Jul 31 14:16:29.793586 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.793568    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"]
Jul 31 14:16:29.796691 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.796673    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"]
Jul 31 14:16:29.813568 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d416104-70a0-4285-bb1c-5ff965a4f9f4-trusted-ca\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.813664 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgl59\" (UniqueName: \"kubernetes.io/projected/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-kube-api-access-fgl59\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:29.813664 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813605    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7355b00f-f3fb-4871-83ee-97769c6c2e6b-serving-cert\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.813664 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksj9w\" (UniqueName: \"kubernetes.io/projected/85ac03c3-dbae-49a4-a406-404da14f2ecd-kube-api-access-ksj9w\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:29.813664 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813658    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzd2p\" (UniqueName: \"kubernetes.io/projected/7355b00f-f3fb-4871-83ee-97769c6c2e6b-kube-api-access-pzd2p\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.813852 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813697    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:29.813852 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813754    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d416104-70a0-4285-bb1c-5ff965a4f9f4-config\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.813852 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7355b00f-f3fb-4871-83ee-97769c6c2e6b-config\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.813852 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813835    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5d416104-70a0-4285-bb1c-5ff965a4f9f4-serving-cert\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.814009 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813853    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:29.814009 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813884    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv45m\" (UniqueName: \"kubernetes.io/projected/5d416104-70a0-4285-bb1c-5ff965a4f9f4-kube-api-access-fv45m\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.814009 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.813900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:29.862957 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.862932    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87"]
Jul 31 14:16:29.864699 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.864684    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87"
Jul 31 14:16:29.866416 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.866397    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q"]
Jul 31 14:16:29.868168 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.868150    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"]
Jul 31 14:16:29.868285 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.868268    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q"
Jul 31 14:16:29.868626 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.868506    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 31 14:16:29.868626 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.868547    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:16:29.868626 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.868561    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7srfp\""
Jul 31 14:16:29.870287 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.870265    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:29.870956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.870939    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-p78s4\""
Jul 31 14:16:29.871073 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.871050    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-lb47s"]
Jul 31 14:16:29.873217 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.873202    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-88648cb45-9npzq"]
Jul 31 14:16:29.873304 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.873293    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:29.873499 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.873482    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:16:29.873776 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.873759    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mwb45\""
Jul 31 14:16:29.873855 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.873818    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 31 14:16:29.873912 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.873878    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 31 14:16:29.873969 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.873955    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 31 14:16:29.875018 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.875005    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.875686 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.875670    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 31 14:16:29.875898 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.875884    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-nwzm8\""
Jul 31 14:16:29.875974 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.875960    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 14:16:29.876053 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.876008    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 31 14:16:29.876566 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.876547    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 14:16:29.877779 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.877734    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 14:16:29.878278 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.877994    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 14:16:29.878278 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.878122    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87"]
Jul 31 14:16:29.878278 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.878125    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 14:16:29.878278 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.878267    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9w89q\""
Jul 31 14:16:29.878777 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.878758    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q"]
Jul 31 14:16:29.881947 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.881923    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"]
Jul 31 14:16:29.886688 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.886663    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 14:16:29.888338 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.888317    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-lb47s"]
Jul 31 14:16:29.891999 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.891980    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 31 14:16:29.896587 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.896569    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-88648cb45-9npzq"]
Jul 31 14:16:29.914461 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:29.914571 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d416104-70a0-4285-bb1c-5ff965a4f9f4-trusted-ca\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.914571 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914489    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-bound-sa-token\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.914571 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzd2p\" (UniqueName: \"kubernetes.io/projected/7355b00f-f3fb-4871-83ee-97769c6c2e6b-kube-api-access-pzd2p\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.914695 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914570    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzh7v\" (UniqueName: \"kubernetes.io/projected/c6e38f19-a6c6-4eb8-909b-df5296e6655e-kube-api-access-mzh7v\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:29.914695 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:29.914648    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:16:29.914759 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:29.914713    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls podName:85ac03c3-dbae-49a4-a406-404da14f2ecd nodeName:}" failed. No retries permitted until 2026-07-31 14:16:30.414697413 +0000 UTC m=+229.287260643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qmsfh" (UID: "85ac03c3-dbae-49a4-a406-404da14f2ecd") : secret "samples-operator-tls" not found
Jul 31 14:16:29.914759 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-trusted-ca\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.914830 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914758    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d648bb3e-dc50-4715-b612-08f683abebd8-ca-trust-extracted\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.914830 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e38f19-a6c6-4eb8-909b-df5296e6655e-trusted-ca-bundle\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:29.914830 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914805    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.914830 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv45m\" (UniqueName: \"kubernetes.io/projected/5d416104-70a0-4285-bb1c-5ff965a4f9f4-kube-api-access-fv45m\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.914982 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914856    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7355b00f-f3fb-4871-83ee-97769c6c2e6b-config\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.914982 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:29.914982 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914944    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5d416104-70a0-4285-bb1c-5ff965a4f9f4-serving-cert\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.915147 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.914981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-registry-certificates\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.915147 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctcj\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-kube-api-access-dctcj\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.915147 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915059    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9dn\" (UniqueName: \"kubernetes.io/projected/04340f30-8169-4d89-94fd-3f09cc420e1f-kube-api-access-jf9dn\") pod \"network-check-source-65bdbbff45-r2p2q\" (UID: \"04340f30-8169-4d89-94fd-3f09cc420e1f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q"
Jul 31 14:16:29.915147 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e38f19-a6c6-4eb8-909b-df5296e6655e-service-ca-bundle\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:29.915339 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-image-registry-private-configuration\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.915339 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915194    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-installation-pull-secrets\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:29.915339 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915239    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgl59\" (UniqueName: \"kubernetes.io/projected/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-kube-api-access-fgl59\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:29.915339 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915268    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aeed834-fbee-468e-ab99-ac7121d9fb08-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:29.915339 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915293    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeed834-fbee-468e-ab99-ac7121d9fb08-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:29.915339 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915320    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e38f19-a6c6-4eb8-909b-df5296e6655e-serving-cert\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:29.915631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915347    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7355b00f-f3fb-4871-83ee-97769c6c2e6b-serving-cert\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.915631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksj9w\" (UniqueName: \"kubernetes.io/projected/85ac03c3-dbae-49a4-a406-404da14f2ecd-kube-api-access-ksj9w\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:29.915631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915401    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6e38f19-a6c6-4eb8-909b-df5296e6655e-tmp\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:29.915631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915424    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7355b00f-f3fb-4871-83ee-97769c6c2e6b-config\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.915631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5d416104-70a0-4285-bb1c-5ff965a4f9f4-trusted-ca\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.915631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915530    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c6e38f19-a6c6-4eb8-909b-df5296e6655e-snapshots\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:29.915631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915561    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:29.915631 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d416104-70a0-4285-bb1c-5ff965a4f9f4-config\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.916011 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915644    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245gb\" (UniqueName: \"kubernetes.io/projected/6b83c90c-18c0-4778-8db6-7f8444419455-kube-api-access-245gb\") pod \"volume-data-source-validator-5b8776669-nft87\" (UID: \"6b83c90c-18c0-4778-8db6-7f8444419455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87"
Jul 31 14:16:29.916011 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915677    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:29.916011 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:29.915688    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:29.916011 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.915693    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbq6q\" (UniqueName: \"kubernetes.io/projected/6aeed834-fbee-468e-ab99-ac7121d9fb08-kube-api-access-jbq6q\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:29.916011 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:29.915752    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls podName:c5a24c7f-fb9e-4ead-b770-68d870cda0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:16:30.415739255 +0000 UTC m=+229.288302489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-87wx8" (UID: "c5a24c7f-fb9e-4ead-b770-68d870cda0ed") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:29.916292 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.916069    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5d416104-70a0-4285-bb1c-5ff965a4f9f4-config\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.917461 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.917442    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5d416104-70a0-4285-bb1c-5ff965a4f9f4-serving-cert\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.918005 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.917989    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7355b00f-f3fb-4871-83ee-97769c6c2e6b-serving-cert\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.922758 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.922738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzd2p\" (UniqueName: \"kubernetes.io/projected/7355b00f-f3fb-4871-83ee-97769c6c2e6b-kube-api-access-pzd2p\") pod \"service-ca-operator-784f97bcf6-5kzfg\" (UID: \"7355b00f-f3fb-4871-83ee-97769c6c2e6b\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:29.923110 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.923091    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv45m\" (UniqueName: \"kubernetes.io/projected/5d416104-70a0-4285-bb1c-5ff965a4f9f4-kube-api-access-fv45m\") pod \"console-operator-5ccd6b445f-ll4lr\" (UID: \"5d416104-70a0-4285-bb1c-5ff965a4f9f4\") " pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:29.925742 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.925721    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksj9w\" (UniqueName: \"kubernetes.io/projected/85ac03c3-dbae-49a4-a406-404da14f2ecd-kube-api-access-ksj9w\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:29.925818 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:29.925788    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgl59\" (UniqueName: \"kubernetes.io/projected/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-kube-api-access-fgl59\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:30.016960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.016923    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245gb\" (UniqueName: \"kubernetes.io/projected/6b83c90c-18c0-4778-8db6-7f8444419455-kube-api-access-245gb\") pod \"volume-data-source-validator-5b8776669-nft87\" (UID: \"6b83c90c-18c0-4778-8db6-7f8444419455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87"
Jul 31 14:16:30.017169 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.016980    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbq6q\" (UniqueName: \"kubernetes.io/projected/6aeed834-fbee-468e-ab99-ac7121d9fb08-kube-api-access-jbq6q\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:30.017169 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-bound-sa-token\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.017169 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzh7v\" (UniqueName: \"kubernetes.io/projected/c6e38f19-a6c6-4eb8-909b-df5296e6655e-kube-api-access-mzh7v\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.017169 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-trusted-ca\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.017169 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d648bb3e-dc50-4715-b612-08f683abebd8-ca-trust-extracted\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.017452 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e38f19-a6c6-4eb8-909b-df5296e6655e-trusted-ca-bundle\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.017452 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017204    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.017452 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.017304    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:16:30.017452 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.017318    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-88648cb45-9npzq: secret "image-registry-tls" not found
Jul 31 14:16:30.017452 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-registry-certificates\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.017452 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctcj\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-kube-api-access-dctcj\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.017452 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.017417    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls podName:d648bb3e-dc50-4715-b612-08f683abebd8 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:30.517398037 +0000 UTC m=+229.389961268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls") pod "image-registry-88648cb45-9npzq" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8") : secret "image-registry-tls" not found
Jul 31 14:16:30.017452 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9dn\" (UniqueName: \"kubernetes.io/projected/04340f30-8169-4d89-94fd-3f09cc420e1f-kube-api-access-jf9dn\") pod \"network-check-source-65bdbbff45-r2p2q\" (UID: \"04340f30-8169-4d89-94fd-3f09cc420e1f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e38f19-a6c6-4eb8-909b-df5296e6655e-service-ca-bundle\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-image-registry-private-configuration\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017546    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-installation-pull-secrets\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017577    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aeed834-fbee-468e-ab99-ac7121d9fb08-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeed834-fbee-468e-ab99-ac7121d9fb08-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e38f19-a6c6-4eb8-909b-df5296e6655e-serving-cert\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6e38f19-a6c6-4eb8-909b-df5296e6655e-tmp\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017693    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c6e38f19-a6c6-4eb8-909b-df5296e6655e-snapshots\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.018004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.017750    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d648bb3e-dc50-4715-b612-08f683abebd8-ca-trust-extracted\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.018435 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.018151    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e38f19-a6c6-4eb8-909b-df5296e6655e-trusted-ca-bundle\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.018435 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.018198    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-registry-certificates\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.018435 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.018211    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6e38f19-a6c6-4eb8-909b-df5296e6655e-service-ca-bundle\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.018435 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.018334    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-trusted-ca\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.018435 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.018401    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c6e38f19-a6c6-4eb8-909b-df5296e6655e-snapshots\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.018435 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.018406    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6e38f19-a6c6-4eb8-909b-df5296e6655e-tmp\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.018665 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.018432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6aeed834-fbee-468e-ab99-ac7121d9fb08-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:30.020153 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.020113    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aeed834-fbee-468e-ab99-ac7121d9fb08-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:30.020412 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.020393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6e38f19-a6c6-4eb8-909b-df5296e6655e-serving-cert\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.020554 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.020539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-installation-pull-secrets\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.020674 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.020653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-image-registry-private-configuration\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.025750 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.025725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-bound-sa-token\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.026239 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.026221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-245gb\" (UniqueName: \"kubernetes.io/projected/6b83c90c-18c0-4778-8db6-7f8444419455-kube-api-access-245gb\") pod \"volume-data-source-validator-5b8776669-nft87\" (UID: \"6b83c90c-18c0-4778-8db6-7f8444419455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87"
Jul 31 14:16:30.026366 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.026344    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbq6q\" (UniqueName: \"kubernetes.io/projected/6aeed834-fbee-468e-ab99-ac7121d9fb08-kube-api-access-jbq6q\") pod \"kube-storage-version-migrator-operator-7878f4b88c-pd68z\" (UID: \"6aeed834-fbee-468e-ab99-ac7121d9fb08\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:30.026633 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.026610    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzh7v\" (UniqueName: \"kubernetes.io/projected/c6e38f19-a6c6-4eb8-909b-df5296e6655e-kube-api-access-mzh7v\") pod \"insights-operator-77696685b-lb47s\" (UID: \"c6e38f19-a6c6-4eb8-909b-df5296e6655e\") " pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.027453 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.027421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9dn\" (UniqueName: \"kubernetes.io/projected/04340f30-8169-4d89-94fd-3f09cc420e1f-kube-api-access-jf9dn\") pod \"network-check-source-65bdbbff45-r2p2q\" (UID: \"04340f30-8169-4d89-94fd-3f09cc420e1f\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q"
Jul 31 14:16:30.027766 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.027748    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctcj\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-kube-api-access-dctcj\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.079593 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.079555    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:30.094630 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.094602    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"
Jul 31 14:16:30.175438 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.175413    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87"
Jul 31 14:16:30.184431 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.184403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q"
Jul 31 14:16:30.193963 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.193784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"
Jul 31 14:16:30.202053 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.201928    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-lb47s"
Jul 31 14:16:30.206522 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.206440    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-ll4lr"]
Jul 31 14:16:30.210904 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:30.210874    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d416104_70a0_4285_bb1c_5ff965a4f9f4.slice/crio-633b08efb3d37a9d5adcfe9ff719d78f2af247021727ae2f600fae97aac1882e WatchSource:0}: Error finding container 633b08efb3d37a9d5adcfe9ff719d78f2af247021727ae2f600fae97aac1882e: Status 404 returned error can't find the container with id 633b08efb3d37a9d5adcfe9ff719d78f2af247021727ae2f600fae97aac1882e
Jul 31 14:16:30.224416 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.224336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" event={"ID":"5d416104-70a0-4285-bb1c-5ff965a4f9f4","Type":"ContainerStarted","Data":"633b08efb3d37a9d5adcfe9ff719d78f2af247021727ae2f600fae97aac1882e"}
Jul 31 14:16:30.225236 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.225074    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg"]
Jul 31 14:16:30.227972 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:30.227914    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7355b00f_f3fb_4871_83ee_97769c6c2e6b.slice/crio-f74c7493fab56c057bdaf58e725798a630a26ec1693c8f41c272a428048040c2 WatchSource:0}: Error finding container f74c7493fab56c057bdaf58e725798a630a26ec1693c8f41c272a428048040c2: Status 404 returned error can't find the container with id f74c7493fab56c057bdaf58e725798a630a26ec1693c8f41c272a428048040c2
Jul 31 14:16:30.329248 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.329217    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87"]
Jul 31 14:16:30.333568 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:30.333041    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b83c90c_18c0_4778_8db6_7f8444419455.slice/crio-e2e73c443307500e93b6956ed2d8f02d1fa70c735afca4a2b893b71feebbbbaa WatchSource:0}: Error finding container e2e73c443307500e93b6956ed2d8f02d1fa70c735afca4a2b893b71feebbbbaa: Status 404 returned error can't find the container with id e2e73c443307500e93b6956ed2d8f02d1fa70c735afca4a2b893b71feebbbbaa
Jul 31 14:16:30.344384 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.344360    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q"]
Jul 31 14:16:30.347953 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:30.347923    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04340f30_8169_4d89_94fd_3f09cc420e1f.slice/crio-7e584376d721a521298f4fa442c85012c75348d9ab9546adaf862c8e1e9dc94c WatchSource:0}: Error finding container 7e584376d721a521298f4fa442c85012c75348d9ab9546adaf862c8e1e9dc94c: Status 404 returned error can't find the container with id 7e584376d721a521298f4fa442c85012c75348d9ab9546adaf862c8e1e9dc94c
Jul 31 14:16:30.361270 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.361246    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z"]
Jul 31 14:16:30.363635 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:30.363603    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeed834_fbee_468e_ab99_ac7121d9fb08.slice/crio-97f2a933ecbd012f706d3ac52d798e177b823eee30d53c2dcd4983b8a593334d WatchSource:0}: Error finding container 97f2a933ecbd012f706d3ac52d798e177b823eee30d53c2dcd4983b8a593334d: Status 404 returned error can't find the container with id 97f2a933ecbd012f706d3ac52d798e177b823eee30d53c2dcd4983b8a593334d
Jul 31 14:16:30.380849 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.380817    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-lb47s"]
Jul 31 14:16:30.384037 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:30.383995    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e38f19_a6c6_4eb8_909b_df5296e6655e.slice/crio-d3d037aaa9d62ddce44f937653f5baaa1f48294afb8731ab5f3bc26a5bf52316 WatchSource:0}: Error finding container d3d037aaa9d62ddce44f937653f5baaa1f48294afb8731ab5f3bc26a5bf52316: Status 404 returned error can't find the container with id d3d037aaa9d62ddce44f937653f5baaa1f48294afb8731ab5f3bc26a5bf52316
Jul 31 14:16:30.420363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.420331    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:30.420532 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.420396    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:30.420532 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.420431    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:30.420532 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.420512    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls podName:c5a24c7f-fb9e-4ead-b770-68d870cda0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:16:31.420490407 +0000 UTC m=+230.293053646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-87wx8" (UID: "c5a24c7f-fb9e-4ead-b770-68d870cda0ed") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:30.420532 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.420520    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:16:30.420745 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.420576    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls podName:85ac03c3-dbae-49a4-a406-404da14f2ecd nodeName:}" failed. No retries permitted until 2026-07-31 14:16:31.420559916 +0000 UTC m=+230.293123169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qmsfh" (UID: "85ac03c3-dbae-49a4-a406-404da14f2ecd") : secret "samples-operator-tls" not found
Jul 31 14:16:30.521183 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:30.521145    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:30.521342 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.521301    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:16:30.521342 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.521316    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-88648cb45-9npzq: secret "image-registry-tls" not found
Jul 31 14:16:30.521406 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:30.521369    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls podName:d648bb3e-dc50-4715-b612-08f683abebd8 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:31.521353705 +0000 UTC m=+230.393916954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls") pod "image-registry-88648cb45-9npzq" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8") : secret "image-registry-tls" not found
Jul 31 14:16:31.229832 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.229760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87" event={"ID":"6b83c90c-18c0-4778-8db6-7f8444419455","Type":"ContainerStarted","Data":"e2e73c443307500e93b6956ed2d8f02d1fa70c735afca4a2b893b71feebbbbaa"}
Jul 31 14:16:31.231312 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.231239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-lb47s" event={"ID":"c6e38f19-a6c6-4eb8-909b-df5296e6655e","Type":"ContainerStarted","Data":"d3d037aaa9d62ddce44f937653f5baaa1f48294afb8731ab5f3bc26a5bf52316"}
Jul 31 14:16:31.233627 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.233585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z" event={"ID":"6aeed834-fbee-468e-ab99-ac7121d9fb08","Type":"ContainerStarted","Data":"97f2a933ecbd012f706d3ac52d798e177b823eee30d53c2dcd4983b8a593334d"}
Jul 31 14:16:31.237047 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.236242    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q" event={"ID":"04340f30-8169-4d89-94fd-3f09cc420e1f","Type":"ContainerStarted","Data":"743951cfbd845de681d98a9e3b89554704803043e442095a163f80635e344a42"}
Jul 31 14:16:31.237047 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.236280    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q" event={"ID":"04340f30-8169-4d89-94fd-3f09cc420e1f","Type":"ContainerStarted","Data":"7e584376d721a521298f4fa442c85012c75348d9ab9546adaf862c8e1e9dc94c"}
Jul 31 14:16:31.238336 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.238260    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg" event={"ID":"7355b00f-f3fb-4871-83ee-97769c6c2e6b","Type":"ContainerStarted","Data":"f74c7493fab56c057bdaf58e725798a630a26ec1693c8f41c272a428048040c2"}
Jul 31 14:16:31.252725 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.251914    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-r2p2q" podStartSLOduration=2.251895373 podStartE2EDuration="2.251895373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:31.25086798 +0000 UTC m=+230.123431234" watchObservedRunningTime="2026-07-31 14:16:31.251895373 +0000 UTC m=+230.124458626"
Jul 31 14:16:31.429790 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.429748    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:31.429967 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.429900    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:31.429967 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:31.429917    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:16:31.430086 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:31.430002    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls podName:85ac03c3-dbae-49a4-a406-404da14f2ecd nodeName:}" failed. No retries permitted until 2026-07-31 14:16:33.429981314 +0000 UTC m=+232.302544567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qmsfh" (UID: "85ac03c3-dbae-49a4-a406-404da14f2ecd") : secret "samples-operator-tls" not found
Jul 31 14:16:31.430086 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:31.430009    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:31.430200 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:31.430086    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls podName:c5a24c7f-fb9e-4ead-b770-68d870cda0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:16:33.430068565 +0000 UTC m=+232.302631797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-87wx8" (UID: "c5a24c7f-fb9e-4ead-b770-68d870cda0ed") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:31.531508 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:31.531352    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:31.531768 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:31.531542    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:16:31.531768 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:31.531567    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-88648cb45-9npzq: secret "image-registry-tls" not found
Jul 31 14:16:31.531768 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:31.531635    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls podName:d648bb3e-dc50-4715-b612-08f683abebd8 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:33.531614604 +0000 UTC m=+232.404177841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls") pod "image-registry-88648cb45-9npzq" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8") : secret "image-registry-tls" not found
Jul 31 14:16:32.244035 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:32.243973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87" event={"ID":"6b83c90c-18c0-4778-8db6-7f8444419455","Type":"ContainerStarted","Data":"e4ee5187dd5dcaf469b90cbe95e3cf0de66a9578b32c4462ec754248b8c88432"}
Jul 31 14:16:32.262328 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:32.261225    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-nft87" podStartSLOduration=1.985930738 podStartE2EDuration="3.261204506s" totalImagesPullingTime="1.275273768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:32.259767538 +0000 UTC m=+231.132330793" watchObservedRunningTime="2026-07-31 14:16:32.261204506 +0000 UTC m=+231.133767761"
Jul 31 14:16:33.250445 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.250104    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg" event={"ID":"7355b00f-f3fb-4871-83ee-97769c6c2e6b","Type":"ContainerStarted","Data":"9a5be18b0b7be7343690151cc588f06ffa23416461719da0f4eb8c388b655bdc"}
Jul 31 14:16:33.252991 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.252679    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/0.log"
Jul 31 14:16:33.252991 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.252719    2558 generic.go:356] "Generic (PLEG): container finished" podID="5d416104-70a0-4285-bb1c-5ff965a4f9f4" containerID="c98c4b9ef508d7474658fa6b7a13307a00db6597fa222126e3f34820ca0261e8" exitCode=255
Jul 31 14:16:33.253207 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.253180    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" event={"ID":"5d416104-70a0-4285-bb1c-5ff965a4f9f4","Type":"ContainerDied","Data":"c98c4b9ef508d7474658fa6b7a13307a00db6597fa222126e3f34820ca0261e8"}
Jul 31 14:16:33.253366 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.253351    2558 scope.go:117] "RemoveContainer" containerID="c98c4b9ef508d7474658fa6b7a13307a00db6597fa222126e3f34820ca0261e8"
Jul 31 14:16:33.266097 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.266056    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg" podStartSLOduration=1.617946603 podStartE2EDuration="4.266041193s" totalImagesPullingTime="2.64809459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:33.264964245 +0000 UTC m=+232.137527501" watchObservedRunningTime="2026-07-31 14:16:33.266041193 +0000 UTC m=+232.138604445"
Jul 31 14:16:33.450541 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.449921    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:33.450541 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.449993    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:33.450541 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:33.450139    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:16:33.450541 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:33.450202    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls podName:85ac03c3-dbae-49a4-a406-404da14f2ecd nodeName:}" failed. No retries permitted until 2026-07-31 14:16:37.450182546 +0000 UTC m=+236.322745783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qmsfh" (UID: "85ac03c3-dbae-49a4-a406-404da14f2ecd") : secret "samples-operator-tls" not found
Jul 31 14:16:33.450851 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:33.450614    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:33.450851 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:33.450680    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls podName:c5a24c7f-fb9e-4ead-b770-68d870cda0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:16:37.450662487 +0000 UTC m=+236.323225731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-87wx8" (UID: "c5a24c7f-fb9e-4ead-b770-68d870cda0ed") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:33.550743 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:33.550704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:33.550920 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:33.550819    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:16:33.550920 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:33.550842    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-88648cb45-9npzq: secret "image-registry-tls" not found
Jul 31 14:16:33.550920 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:33.550906    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls podName:d648bb3e-dc50-4715-b612-08f683abebd8 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:37.550889842 +0000 UTC m=+236.423453077 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls") pod "image-registry-88648cb45-9npzq" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8") : secret "image-registry-tls" not found
Jul 31 14:16:34.257795 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:34.257720    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/1.log"
Jul 31 14:16:34.258268 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:34.258173    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/0.log"
Jul 31 14:16:34.258268 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:34.258211    2558 generic.go:356] "Generic (PLEG): container finished" podID="5d416104-70a0-4285-bb1c-5ff965a4f9f4" containerID="a3de26c807c6d222c7c3654d6d7948e05155e651b7f6cf160cfab6320a8c08e9" exitCode=255
Jul 31 14:16:34.258377 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:34.258299    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" event={"ID":"5d416104-70a0-4285-bb1c-5ff965a4f9f4","Type":"ContainerDied","Data":"a3de26c807c6d222c7c3654d6d7948e05155e651b7f6cf160cfab6320a8c08e9"}
Jul 31 14:16:34.258377 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:34.258368    2558 scope.go:117] "RemoveContainer" containerID="c98c4b9ef508d7474658fa6b7a13307a00db6597fa222126e3f34820ca0261e8"
Jul 31 14:16:34.258594 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:34.258577    2558 scope.go:117] "RemoveContainer" containerID="a3de26c807c6d222c7c3654d6d7948e05155e651b7f6cf160cfab6320a8c08e9"
Jul 31 14:16:34.258811 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:34.258777    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-ll4lr_openshift-console-operator(5d416104-70a0-4285-bb1c-5ff965a4f9f4)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" podUID="5d416104-70a0-4285-bb1c-5ff965a4f9f4"
Jul 31 14:16:35.174617 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.174588    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d8c7b"]
Jul 31 14:16:35.178032 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.178002    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.180723 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.180694    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 14:16:35.180829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.180795    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t7s2z\""
Jul 31 14:16:35.181901 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.181877    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 14:16:35.190176 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.190156    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d8c7b"]
Jul 31 14:16:35.263698 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.263671    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/1.log"
Jul 31 14:16:35.264156 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.264082    2558 scope.go:117] "RemoveContainer" containerID="a3de26c807c6d222c7c3654d6d7948e05155e651b7f6cf160cfab6320a8c08e9"
Jul 31 14:16:35.264279 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.264254    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-ll4lr_openshift-console-operator(5d416104-70a0-4285-bb1c-5ff965a4f9f4)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" podUID="5d416104-70a0-4285-bb1c-5ff965a4f9f4"
Jul 31 14:16:35.264465 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.264435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.264585 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.264525    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-data-volume\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.264647 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.264594    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g98cz\" (UniqueName: \"kubernetes.io/projected/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-api-access-g98cz\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.264647 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.264624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.264738 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.264660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-crio-socket\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.265451 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.265429    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-lb47s" event={"ID":"c6e38f19-a6c6-4eb8-909b-df5296e6655e","Type":"ContainerStarted","Data":"a6ff87080eeb0b3e6da55dfe871fa59f5bf8f8f1de7262f71141fe1fe800c234"}
Jul 31 14:16:35.266884 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.266865    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z" event={"ID":"6aeed834-fbee-468e-ab99-ac7121d9fb08","Type":"ContainerStarted","Data":"45fe8200a2886398d3eac544334a217eab86260d395c6a846a36e5376f185e6f"}
Jul 31 14:16:35.294255 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.294217    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-lb47s" podStartSLOduration=2.20674272 podStartE2EDuration="6.294203609s" totalImagesPullingTime="4.087460889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:35.293979186 +0000 UTC m=+234.166542439" watchObservedRunningTime="2026-07-31 14:16:35.294203609 +0000 UTC m=+234.166766861"
Jul 31 14:16:35.308517 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.308474    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z" podStartSLOduration=2.195065222 podStartE2EDuration="6.308460858s" totalImagesPullingTime="4.113395636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:35.307953328 +0000 UTC m=+234.180516583" watchObservedRunningTime="2026-07-31 14:16:35.308460858 +0000 UTC m=+234.181024110"
Jul 31 14:16:35.365515 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.365481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-data-volume\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.365703 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.365629    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g98cz\" (UniqueName: \"kubernetes.io/projected/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-api-access-g98cz\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.365703 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.365680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.365822 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.365736    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-crio-socket\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.365875 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.365832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-data-volume\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.365875 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.365848    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:35.365968 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.365898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.365968 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.365918    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:35.865897132 +0000 UTC m=+234.738460372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:35.366090 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.365966    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:16:35.366090 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.366046    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:35.866006418 +0000 UTC m=+234.738569649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : configmap "kube-rbac-proxy" not found
Jul 31 14:16:35.366090 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.365848    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-crio-socket\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.374755 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.374735    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g98cz\" (UniqueName: \"kubernetes.io/projected/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-api-access-g98cz\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.567861 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.567822    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm"]
Jul 31 14:16:35.571092 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.571076    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm"
Jul 31 14:16:35.573837 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.573809    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 31 14:16:35.573971 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.573894    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-85m6s\""
Jul 31 14:16:35.573971 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.573910    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 31 14:16:35.578506 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.578456    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm"]
Jul 31 14:16:35.667895 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.667861    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btm8\" (UniqueName: \"kubernetes.io/projected/ea8f38c1-1fef-4593-bf10-f1f954c384ce-kube-api-access-4btm8\") pod \"migrator-68bc745d87-z9jlm\" (UID: \"ea8f38c1-1fef-4593-bf10-f1f954c384ce\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm"
Jul 31 14:16:35.769287 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.769252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btm8\" (UniqueName: \"kubernetes.io/projected/ea8f38c1-1fef-4593-bf10-f1f954c384ce-kube-api-access-4btm8\") pod \"migrator-68bc745d87-z9jlm\" (UID: \"ea8f38c1-1fef-4593-bf10-f1f954c384ce\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm"
Jul 31 14:16:35.777185 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.777158    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btm8\" (UniqueName: \"kubernetes.io/projected/ea8f38c1-1fef-4593-bf10-f1f954c384ce-kube-api-access-4btm8\") pod \"migrator-68bc745d87-z9jlm\" (UID: \"ea8f38c1-1fef-4593-bf10-f1f954c384ce\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm"
Jul 31 14:16:35.869922 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.869818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.870101 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.869954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:35.870101 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.869979    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:35.870101 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.870069    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:36.870049525 +0000 UTC m=+235.742612776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:35.870225 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.870100    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:16:35.870225 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:35.870153    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:36.870140231 +0000 UTC m=+235.742703462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : configmap "kube-rbac-proxy" not found
Jul 31 14:16:35.881743 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:35.881719    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm"
Jul 31 14:16:36.002510 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.002480    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm"]
Jul 31 14:16:36.004875 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:36.004845    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8f38c1_1fef_4593_bf10_f1f954c384ce.slice/crio-2252925455ffc42d31cb6d795a17273867edb8a73eb1e2d77bd4a1ffc7dd656b WatchSource:0}: Error finding container 2252925455ffc42d31cb6d795a17273867edb8a73eb1e2d77bd4a1ffc7dd656b: Status 404 returned error can't find the container with id 2252925455ffc42d31cb6d795a17273867edb8a73eb1e2d77bd4a1ffc7dd656b
Jul 31 14:16:36.271017 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.270930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm" event={"ID":"ea8f38c1-1fef-4593-bf10-f1f954c384ce","Type":"ContainerStarted","Data":"2252925455ffc42d31cb6d795a17273867edb8a73eb1e2d77bd4a1ffc7dd656b"}
Jul 31 14:16:36.432819 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.432784    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gddlj"]
Jul 31 14:16:36.437016 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.436995    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.439621 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.439600    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 31 14:16:36.439712 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.439632    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 31 14:16:36.439802 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.439784    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-px6z7\""
Jul 31 14:16:36.439853 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.439788    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 31 14:16:36.439936 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.439922    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 31 14:16:36.443155 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.443132    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gddlj"]
Jul 31 14:16:36.475435 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.475407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcad45cf-faec-44e5-b9d8-8082aef368c6-signing-cabundle\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.475573 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.475456    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b255v\" (UniqueName: \"kubernetes.io/projected/fcad45cf-faec-44e5-b9d8-8082aef368c6-kube-api-access-b255v\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.475573 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.475524    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcad45cf-faec-44e5-b9d8-8082aef368c6-signing-key\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.576170 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.576140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcad45cf-faec-44e5-b9d8-8082aef368c6-signing-cabundle\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.576315 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.576187    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b255v\" (UniqueName: \"kubernetes.io/projected/fcad45cf-faec-44e5-b9d8-8082aef368c6-kube-api-access-b255v\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.576315 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.576215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcad45cf-faec-44e5-b9d8-8082aef368c6-signing-key\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.576799 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.576775    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcad45cf-faec-44e5-b9d8-8082aef368c6-signing-cabundle\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.578538 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.578515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcad45cf-faec-44e5-b9d8-8082aef368c6-signing-key\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.584649 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.584627    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b255v\" (UniqueName: \"kubernetes.io/projected/fcad45cf-faec-44e5-b9d8-8082aef368c6-kube-api-access-b255v\") pod \"service-ca-98c88fbb5-gddlj\" (UID: \"fcad45cf-faec-44e5-b9d8-8082aef368c6\") " pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.747230 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.747187    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-gddlj"
Jul 31 14:16:36.879375 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.879343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:36.879556 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:36.879485    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:16:36.879556 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.879535    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:36.879556 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:36.879556    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:38.87953634 +0000 UTC m=+237.752099573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : configmap "kube-rbac-proxy" not found
Jul 31 14:16:36.879723 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:36.879621    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:36.879723 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:36.879674    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:38.879660073 +0000 UTC m=+237.752223304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:36.883558 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:36.883535    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-gddlj"]
Jul 31 14:16:36.885788 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:36.885767    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcad45cf_faec_44e5_b9d8_8082aef368c6.slice/crio-e7b924409fd9491eb7572f687b4fc0880a4383824d838cc4d1256a761b015e39 WatchSource:0}: Error finding container e7b924409fd9491eb7572f687b4fc0880a4383824d838cc4d1256a761b015e39: Status 404 returned error can't find the container with id e7b924409fd9491eb7572f687b4fc0880a4383824d838cc4d1256a761b015e39
Jul 31 14:16:37.274685 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:37.274659    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-gddlj" event={"ID":"fcad45cf-faec-44e5-b9d8-8082aef368c6","Type":"ContainerStarted","Data":"621a265ba54846e970666d2e3eeea55edf3ec734785c7a80a641da5d9d8e3fb6"}
Jul 31 14:16:37.275056 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:37.274716    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-gddlj" event={"ID":"fcad45cf-faec-44e5-b9d8-8082aef368c6","Type":"ContainerStarted","Data":"e7b924409fd9491eb7572f687b4fc0880a4383824d838cc4d1256a761b015e39"}
Jul 31 14:16:37.292231 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:37.292118    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-gddlj" podStartSLOduration=1.29209932 podStartE2EDuration="1.29209932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:37.29174892 +0000 UTC m=+236.164312174" watchObservedRunningTime="2026-07-31 14:16:37.29209932 +0000 UTC m=+236.164662552"
Jul 31 14:16:37.339654 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:37.339601    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rtc4t_57c7c983-8b4d-4494-be24-c63071b5ad0a/dns-node-resolver/0.log"
Jul 31 14:16:37.486056 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:37.486001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:37.486217 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:37.486070    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:37.486217 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:37.486175    2558 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:37.486217 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:37.486190    2558 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 31 14:16:37.486351 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:37.486241    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls podName:85ac03c3-dbae-49a4-a406-404da14f2ecd nodeName:}" failed. No retries permitted until 2026-07-31 14:16:45.486227993 +0000 UTC m=+244.358791224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qmsfh" (UID: "85ac03c3-dbae-49a4-a406-404da14f2ecd") : secret "samples-operator-tls" not found
Jul 31 14:16:37.486351 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:37.486253    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls podName:c5a24c7f-fb9e-4ead-b770-68d870cda0ed nodeName:}" failed. No retries permitted until 2026-07-31 14:16:45.486247609 +0000 UTC m=+244.358810839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-87wx8" (UID: "c5a24c7f-fb9e-4ead-b770-68d870cda0ed") : secret "cluster-monitoring-operator-tls" not found
Jul 31 14:16:37.586434 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:37.586389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:37.586615 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:37.586522    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:16:37.586615 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:37.586540    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-88648cb45-9npzq: secret "image-registry-tls" not found
Jul 31 14:16:37.586615 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:37.586603    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls podName:d648bb3e-dc50-4715-b612-08f683abebd8 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:45.586584794 +0000 UTC m=+244.459148038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls") pod "image-registry-88648cb45-9npzq" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8") : secret "image-registry-tls" not found
Jul 31 14:16:37.939245 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:37.939168    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gkw4n_66f05e4e-0197-4037-923c-c626b7982ab2/node-ca/0.log"
Jul 31 14:16:38.281204 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:38.281054    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm" event={"ID":"ea8f38c1-1fef-4593-bf10-f1f954c384ce","Type":"ContainerStarted","Data":"1a6de5f54a64917d7d1bd39e7eb62e8a1efaa79957a0651d0d903e50f1d0dc80"}
Jul 31 14:16:38.281204 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:38.281117    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm" event={"ID":"ea8f38c1-1fef-4593-bf10-f1f954c384ce","Type":"ContainerStarted","Data":"f085193854fd5106d306f3192f3cce1e150df72f49fe641b11baa0247f008200"}
Jul 31 14:16:38.301429 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:38.301374    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-z9jlm" podStartSLOduration=2.055358025 podStartE2EDuration="3.301354465s" totalImagesPullingTime="1.24599644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:38.300308892 +0000 UTC m=+237.172872150" watchObservedRunningTime="2026-07-31 14:16:38.301354465 +0000 UTC m=+237.173917723"
Jul 31 14:16:38.899106 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:38.899069    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:38.899284 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:38.899179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:38.899284 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:38.899204    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:16:38.899378 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:38.899287    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:42.899266711 +0000 UTC m=+241.771829959 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : configmap "kube-rbac-proxy" not found
Jul 31 14:16:38.899378 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:38.899299    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:38.899378 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:38.899349    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:42.8993346 +0000 UTC m=+241.771897838 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:39.138591 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:39.138561    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m98v2_96a9dc2b-3f07-4be6-85a0-722f0b87c5ac/serve-healthcheck-canary/0.log"
Jul 31 14:16:40.080273 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:40.080180    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:40.080273 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:40.080247    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:16:40.080781 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:40.080690    2558 scope.go:117] "RemoveContainer" containerID="a3de26c807c6d222c7c3654d6d7948e05155e651b7f6cf160cfab6320a8c08e9"
Jul 31 14:16:40.080909 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:40.080884    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-ll4lr_openshift-console-operator(5d416104-70a0-4285-bb1c-5ff965a4f9f4)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" podUID="5d416104-70a0-4285-bb1c-5ff965a4f9f4"
Jul 31 14:16:42.939279 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:42.939240    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:42.939660 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:42.939325    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:42.939660 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:42.939388    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:42.939660 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:42.939421    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:16:42.939660 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:42.939457    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:50.939442637 +0000 UTC m=+249.812005873 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:16:42.939660 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:42.939479    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm podName:e366b3eb-b4bf-48a4-a97d-070b2b2c1977 nodeName:}" failed. No retries permitted until 2026-07-31 14:16:50.939472443 +0000 UTC m=+249.812035674 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d8c7b" (UID: "e366b3eb-b4bf-48a4-a97d-070b2b2c1977") : configmap "kube-rbac-proxy" not found
Jul 31 14:16:45.562039 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.561988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:45.562417 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.562108    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:45.564391 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.564363    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5a24c7f-fb9e-4ead-b770-68d870cda0ed-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-87wx8\" (UID: \"c5a24c7f-fb9e-4ead-b770-68d870cda0ed\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:45.564497 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.564433    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/85ac03c3-dbae-49a4-a406-404da14f2ecd-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qmsfh\" (UID: \"85ac03c3-dbae-49a4-a406-404da14f2ecd\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:45.662666 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.662631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:45.664922 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.664891    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"image-registry-88648cb45-9npzq\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") " pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:45.705134 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.705108    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-nlvv2\""
Jul 31 14:16:45.708785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.708770    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-26w8d\""
Jul 31 14:16:45.712931 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.712917    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"
Jul 31 14:16:45.716804 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.716784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"
Jul 31 14:16:45.807963 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.807939    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9w89q\""
Jul 31 14:16:45.816214 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.816141    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:45.851167 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.851098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8"]
Jul 31 14:16:45.853935 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:45.853901    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a24c7f_fb9e_4ead_b770_68d870cda0ed.slice/crio-a22abb531f34b3660e59d6e1580787457ea8a6760e06b5b45e55faf41180e1d0 WatchSource:0}: Error finding container a22abb531f34b3660e59d6e1580787457ea8a6760e06b5b45e55faf41180e1d0: Status 404 returned error can't find the container with id a22abb531f34b3660e59d6e1580787457ea8a6760e06b5b45e55faf41180e1d0
Jul 31 14:16:45.863163 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.863090    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh"]
Jul 31 14:16:45.979232 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:45.979207    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-88648cb45-9npzq"]
Jul 31 14:16:45.980917 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:45.980889    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd648bb3e_dc50_4715_b612_08f683abebd8.slice/crio-93c0f00e26f2b722c039a28d571e4f1e665f32a97e8d5b8e1606818088e36c2e WatchSource:0}: Error finding container 93c0f00e26f2b722c039a28d571e4f1e665f32a97e8d5b8e1606818088e36c2e: Status 404 returned error can't find the container with id 93c0f00e26f2b722c039a28d571e4f1e665f32a97e8d5b8e1606818088e36c2e
Jul 31 14:16:46.312169 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:46.312133    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh" event={"ID":"85ac03c3-dbae-49a4-a406-404da14f2ecd","Type":"ContainerStarted","Data":"bcbc30d019e18eb5106b5454118964fc4fa708734dd4892bca5da7337527c169"}
Jul 31 14:16:46.313364 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:46.313329    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8" event={"ID":"c5a24c7f-fb9e-4ead-b770-68d870cda0ed","Type":"ContainerStarted","Data":"a22abb531f34b3660e59d6e1580787457ea8a6760e06b5b45e55faf41180e1d0"}
Jul 31 14:16:46.314775 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:46.314752    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-88648cb45-9npzq" event={"ID":"d648bb3e-dc50-4715-b612-08f683abebd8","Type":"ContainerStarted","Data":"0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2"}
Jul 31 14:16:46.314865 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:46.314795    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-88648cb45-9npzq" event={"ID":"d648bb3e-dc50-4715-b612-08f683abebd8","Type":"ContainerStarted","Data":"93c0f00e26f2b722c039a28d571e4f1e665f32a97e8d5b8e1606818088e36c2e"}
Jul 31 14:16:46.314921 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:46.314903    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:16:46.334560 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:46.334504    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-88648cb45-9npzq" podStartSLOduration=17.334484387 podStartE2EDuration="17.334484387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:46.332430657 +0000 UTC m=+245.204993913" watchObservedRunningTime="2026-07-31 14:16:46.334484387 +0000 UTC m=+245.207047640"
Jul 31 14:16:48.325940 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:48.325899    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh" event={"ID":"85ac03c3-dbae-49a4-a406-404da14f2ecd","Type":"ContainerStarted","Data":"57a0a9d8f0fb5cdb2c1e4ed1f2acf147e7be1ccedc183e0d23cb16fd46c91fa8"}
Jul 31 14:16:48.326311 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:48.326000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh" event={"ID":"85ac03c3-dbae-49a4-a406-404da14f2ecd","Type":"ContainerStarted","Data":"5ce7c8c3e9e661ee37675e289b8ecc9fffe8503c6326240d119f6eccfe3df245"}
Jul 31 14:16:48.327528 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:48.327425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8" event={"ID":"c5a24c7f-fb9e-4ead-b770-68d870cda0ed","Type":"ContainerStarted","Data":"0af27c8694a29075d5896b779f1ffe8c8ef1080d7d2e1e047083ccb1f2fba09e"}
Jul 31 14:16:48.342179 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:48.342130    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qmsfh" podStartSLOduration=17.104953303 podStartE2EDuration="19.342115011s" totalImagesPullingTime="2.237161708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:48.341008307 +0000 UTC m=+247.213571573" watchObservedRunningTime="2026-07-31 14:16:48.342115011 +0000 UTC m=+247.214678304"
Jul 31 14:16:48.356487 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:48.356421    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-87wx8" podStartSLOduration=17.050611965 podStartE2EDuration="19.356401177s" totalImagesPullingTime="2.305789212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:48.355322136 +0000 UTC m=+247.227885390" watchObservedRunningTime="2026-07-31 14:16:48.356401177 +0000 UTC m=+247.228964431"
Jul 31 14:16:51.007611 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.007552    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:51.007997 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.007643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:51.008135 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.008115    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:51.009869 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.009852    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e366b3eb-b4bf-48a4-a97d-070b2b2c1977-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8c7b\" (UID: \"e366b3eb-b4bf-48a4-a97d-070b2b2c1977\") " pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:51.089771 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.089736    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t7s2z\""
Jul 31 14:16:51.097758 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.097737    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8c7b"
Jul 31 14:16:51.220779 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.220755    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d8c7b"]
Jul 31 14:16:51.222901 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:51.222875    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode366b3eb_b4bf_48a4_a97d_070b2b2c1977.slice/crio-4f668d2e1562375d8c429592ad2e64914e2a56cf7c0801308307e3a9983c9808 WatchSource:0}: Error finding container 4f668d2e1562375d8c429592ad2e64914e2a56cf7c0801308307e3a9983c9808: Status 404 returned error can't find the container with id 4f668d2e1562375d8c429592ad2e64914e2a56cf7c0801308307e3a9983c9808
Jul 31 14:16:51.338368 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.338336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8c7b" event={"ID":"e366b3eb-b4bf-48a4-a97d-070b2b2c1977","Type":"ContainerStarted","Data":"a6e13ec297ea1086b38c60228c11bd91ee2c2062364f4a1019cd71a7864502b3"}
Jul 31 14:16:51.338501 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:51.338391    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8c7b" event={"ID":"e366b3eb-b4bf-48a4-a97d-070b2b2c1977","Type":"ContainerStarted","Data":"4f668d2e1562375d8c429592ad2e64914e2a56cf7c0801308307e3a9983c9808"}
Jul 31 14:16:52.347139 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:52.347102    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8c7b" event={"ID":"e366b3eb-b4bf-48a4-a97d-070b2b2c1977","Type":"ContainerStarted","Data":"415dd7a452a59546314b1ff55cd4c219bf6ed99c85b0a02f562d4bb6c0d77b41"}
Jul 31 14:16:53.429164 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:53.429135    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:16:53.431351 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:53.431322    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13d90876-b172-48d8-95e3-420e703c71cf-metrics-certs\") pod \"network-metrics-daemon-jbbns\" (UID: \"13d90876-b172-48d8-95e3-420e703c71cf\") " pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:16:53.574072 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:53.574045    2558 scope.go:117] "RemoveContainer" containerID="a3de26c807c6d222c7c3654d6d7948e05155e651b7f6cf160cfab6320a8c08e9"
Jul 31 14:16:53.576722 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:53.576659    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7lwbh\""
Jul 31 14:16:53.584966 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:53.584935    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbbns"
Jul 31 14:16:53.714308 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:53.714283    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jbbns"]
Jul 31 14:16:53.741795 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:53.741764    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d90876_b172_48d8_95e3_420e703c71cf.slice/crio-5308d56fc627afce5d171aeec0d61d8cef0b331be69b68cba0be08fe66e473b4 WatchSource:0}: Error finding container 5308d56fc627afce5d171aeec0d61d8cef0b331be69b68cba0be08fe66e473b4: Status 404 returned error can't find the container with id 5308d56fc627afce5d171aeec0d61d8cef0b331be69b68cba0be08fe66e473b4
Jul 31 14:16:54.355910 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.355881    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:16:54.356317 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.356300    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/1.log"
Jul 31 14:16:54.356405 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.356337    2558 generic.go:356] "Generic (PLEG): container finished" podID="5d416104-70a0-4285-bb1c-5ff965a4f9f4" containerID="c47ab862284df41613f54e0ac0c0b9fba2735d5589ef83028c565dee03288f35" exitCode=255
Jul 31 14:16:54.356452 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.356414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" event={"ID":"5d416104-70a0-4285-bb1c-5ff965a4f9f4","Type":"ContainerDied","Data":"c47ab862284df41613f54e0ac0c0b9fba2735d5589ef83028c565dee03288f35"}
Jul 31 14:16:54.356483 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.356468    2558 scope.go:117] "RemoveContainer" containerID="a3de26c807c6d222c7c3654d6d7948e05155e651b7f6cf160cfab6320a8c08e9"
Jul 31 14:16:54.356831 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.356811    2558 scope.go:117] "RemoveContainer" containerID="c47ab862284df41613f54e0ac0c0b9fba2735d5589ef83028c565dee03288f35"
Jul 31 14:16:54.357033 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:16:54.356999    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-ll4lr_openshift-console-operator(5d416104-70a0-4285-bb1c-5ff965a4f9f4)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" podUID="5d416104-70a0-4285-bb1c-5ff965a4f9f4"
Jul 31 14:16:54.358585 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.358460    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8c7b" event={"ID":"e366b3eb-b4bf-48a4-a97d-070b2b2c1977","Type":"ContainerStarted","Data":"804eb29f078edbf063697a273a0f173ca4b85133dec710fc60c4980db1013b7f"}
Jul 31 14:16:54.359564 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.359547    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbbns" event={"ID":"13d90876-b172-48d8-95e3-420e703c71cf","Type":"ContainerStarted","Data":"5308d56fc627afce5d171aeec0d61d8cef0b331be69b68cba0be08fe66e473b4"}
Jul 31 14:16:54.394106 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:54.394060    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d8c7b" podStartSLOduration=17.41040188 podStartE2EDuration="19.394042565s" totalImagesPullingTime="1.983640685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:16:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:54.39333599 +0000 UTC m=+253.265899244" watchObservedRunningTime="2026-07-31 14:16:54.394042565 +0000 UTC m=+253.266605815"
Jul 31 14:16:55.365275 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:55.365237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbbns" event={"ID":"13d90876-b172-48d8-95e3-420e703c71cf","Type":"ContainerStarted","Data":"51921391e3de489c0e4acbd8cf4696bca1dfb17dc621e80dd914d77c53be7875"}
Jul 31 14:16:55.365670 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:55.365295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbbns" event={"ID":"13d90876-b172-48d8-95e3-420e703c71cf","Type":"ContainerStarted","Data":"a5ad69099926b1c98fd37cc7dff21e32213f219ba2d449182edd09b0d5f5d04c"}
Jul 31 14:16:55.366688 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:55.366665    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:16:55.381136 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:55.381097    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jbbns" podStartSLOduration=253.177308341 podStartE2EDuration="4m14.381081426s" totalImagesPullingTime="1.203773085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:55.379512032 +0000 UTC m=+254.252075290" watchObservedRunningTime="2026-07-31 14:16:55.381081426 +0000 UTC m=+254.253644678"
Jul 31 14:16:58.620305 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.620270    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-88648cb45-9npzq"]
Jul 31 14:16:58.657139 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.657110    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bcc48f99d-w2dpl"]
Jul 31 14:16:58.660347 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.660324    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.672791 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.672769    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bcc48f99d-w2dpl"]
Jul 31 14:16:58.767784 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.767747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b2cc95b-de81-47cf-a8d2-673a511e70fd-image-registry-private-configuration\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.767944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.767790    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b2cc95b-de81-47cf-a8d2-673a511e70fd-ca-trust-extracted\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.767944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.767809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-bound-sa-token\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.767944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.767863    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-registry-tls\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.767944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.767902    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b2cc95b-de81-47cf-a8d2-673a511e70fd-trusted-ca\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.767944 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.767920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b2cc95b-de81-47cf-a8d2-673a511e70fd-installation-pull-secrets\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.768150 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.767956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b2cc95b-de81-47cf-a8d2-673a511e70fd-registry-certificates\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.768150 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.767973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6jk\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-kube-api-access-kb6jk\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868343 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b2cc95b-de81-47cf-a8d2-673a511e70fd-image-registry-private-configuration\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868524 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b2cc95b-de81-47cf-a8d2-673a511e70fd-ca-trust-extracted\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868524 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-bound-sa-token\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868524 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-registry-tls\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868524 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868450    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b2cc95b-de81-47cf-a8d2-673a511e70fd-trusted-ca\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868524 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b2cc95b-de81-47cf-a8d2-673a511e70fd-installation-pull-secrets\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868524 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868507    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b2cc95b-de81-47cf-a8d2-673a511e70fd-registry-certificates\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868814 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868530    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6jk\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-kube-api-access-kb6jk\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.868814 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.868782    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b2cc95b-de81-47cf-a8d2-673a511e70fd-ca-trust-extracted\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.869433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.869408    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b2cc95b-de81-47cf-a8d2-673a511e70fd-registry-certificates\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.869555 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.869511    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b2cc95b-de81-47cf-a8d2-673a511e70fd-trusted-ca\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.870960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.870916    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-registry-tls\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.871324 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.871305    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b2cc95b-de81-47cf-a8d2-673a511e70fd-installation-pull-secrets\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.871471 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.871441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b2cc95b-de81-47cf-a8d2-673a511e70fd-image-registry-private-configuration\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.875874 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.875854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-bound-sa-token\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.875945 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.875918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6jk\" (UniqueName: \"kubernetes.io/projected/6b2cc95b-de81-47cf-a8d2-673a511e70fd-kube-api-access-kb6jk\") pod \"image-registry-5bcc48f99d-w2dpl\" (UID: \"6b2cc95b-de81-47cf-a8d2-673a511e70fd\") " pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:58.969494 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:58.969462    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:59.098930 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:59.098894    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bcc48f99d-w2dpl"]
Jul 31 14:16:59.101720 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:16:59.101687    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2cc95b_de81_47cf_a8d2_673a511e70fd.slice/crio-ea28b53b2eff119dfcc04730422ce61d4f781f028ad999026282c0dd2744b1f2 WatchSource:0}: Error finding container ea28b53b2eff119dfcc04730422ce61d4f781f028ad999026282c0dd2744b1f2: Status 404 returned error can't find the container with id ea28b53b2eff119dfcc04730422ce61d4f781f028ad999026282c0dd2744b1f2
Jul 31 14:16:59.384133 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:59.384093    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl" event={"ID":"6b2cc95b-de81-47cf-a8d2-673a511e70fd","Type":"ContainerStarted","Data":"09a88679add8e205dd76e6f346633e65226120a1269023e6679e04c58b8bce5b"}
Jul 31 14:16:59.384319 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:59.384152    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl" event={"ID":"6b2cc95b-de81-47cf-a8d2-673a511e70fd","Type":"ContainerStarted","Data":"ea28b53b2eff119dfcc04730422ce61d4f781f028ad999026282c0dd2744b1f2"}
Jul 31 14:16:59.384319 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:59.384188    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:16:59.403587 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:16:59.403480    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl" podStartSLOduration=1.403463665 podStartE2EDuration="1.403463665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:16:59.401936683 +0000 UTC m=+258.274499939" watchObservedRunningTime="2026-07-31 14:16:59.403463665 +0000 UTC m=+258.276026923"
Jul 31 14:17:00.079793 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:00.079755    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:17:00.080182 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:00.079815    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:17:00.080182 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:00.080146    2558 scope.go:117] "RemoveContainer" containerID="c47ab862284df41613f54e0ac0c0b9fba2735d5589ef83028c565dee03288f35"
Jul 31 14:17:00.080312 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:17:00.080293    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-ll4lr_openshift-console-operator(5d416104-70a0-4285-bb1c-5ff965a4f9f4)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" podUID="5d416104-70a0-4285-bb1c-5ff965a4f9f4"
Jul 31 14:17:08.089162 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.089077    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kfhj9"]
Jul 31 14:17:08.131650 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.131619    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.134596 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.134573    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 14:17:08.134708 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.134623    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 14:17:08.134778 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.134761    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 14:17:08.134962 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.134873    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zn2sg\""
Jul 31 14:17:08.135822 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.135804    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 14:17:08.142304 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-sys\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.142402 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142313    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-textfile\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.142402 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142337    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-root\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.142496 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142441    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-wtmp\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.142496 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142479    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-metrics-client-ca\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.142577 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142510    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.142577 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142555    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-tls\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.142653 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.142693 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.142658    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2jc\" (UniqueName: \"kubernetes.io/projected/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-kube-api-access-2h2jc\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243412 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2jc\" (UniqueName: \"kubernetes.io/projected/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-kube-api-access-2h2jc\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-sys\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243444    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-textfile\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243463    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-root\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243506    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-wtmp\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243508    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-sys\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-metrics-client-ca\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243550    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243585    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-root\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243605 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-tls\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243939 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243939 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243664    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-wtmp\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.243939 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.243798    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-textfile\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.244137 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.244118    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-metrics-client-ca\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.244196 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.244154    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-accelerators-collector-config\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.245986 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.245970    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.246123 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.246105    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-node-exporter-tls\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.250447 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.250426    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2jc\" (UniqueName: \"kubernetes.io/projected/99cb1324-7f9a-45a4-9c2b-be4fc936ad44-kube-api-access-2h2jc\") pod \"node-exporter-kfhj9\" (UID: \"99cb1324-7f9a-45a4-9c2b-be4fc936ad44\") " pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.443519 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.443443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kfhj9"
Jul 31 14:17:08.474695 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:17:08.474663    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99cb1324_7f9a_45a4_9c2b_be4fc936ad44.slice/crio-ed86ddc77ea73212ce29b7752e090e326aa6f1ebebae3dfac73f3edb230a10e1 WatchSource:0}: Error finding container ed86ddc77ea73212ce29b7752e090e326aa6f1ebebae3dfac73f3edb230a10e1: Status 404 returned error can't find the container with id ed86ddc77ea73212ce29b7752e090e326aa6f1ebebae3dfac73f3edb230a10e1
Jul 31 14:17:08.625276 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.625241    2558 patch_prober.go:28] interesting pod/image-registry-88648cb45-9npzq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 31 14:17:08.625422 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:08.625301    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-88648cb45-9npzq" podUID="d648bb3e-dc50-4715-b612-08f683abebd8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:17:09.421282 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:09.421190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfhj9" event={"ID":"99cb1324-7f9a-45a4-9c2b-be4fc936ad44","Type":"ContainerStarted","Data":"ed86ddc77ea73212ce29b7752e090e326aa6f1ebebae3dfac73f3edb230a10e1"}
Jul 31 14:17:10.063069 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.063040    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57889dc56-7n8pq"]
Jul 31 14:17:10.092947 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.092920    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57889dc56-7n8pq"]
Jul 31 14:17:10.093104 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.093078    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.095885 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.095861    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 31 14:17:10.096015 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.095885    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 31 14:17:10.096015 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.095985    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 31 14:17:10.096015 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.095885    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-99cs0avucpe35\""
Jul 31 14:17:10.096299 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.096282    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 31 14:17:10.096382 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.096362    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 31 14:17:10.096447 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.096425    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-776hw\""
Jul 31 14:17:10.157112 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.157082    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.157261 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.157124    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.157261 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.157173    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.157261 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.157197    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-grpc-tls\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.157261 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.157235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624e76da-697c-4deb-8d42-12eecc3ff4a7-metrics-client-ca\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.157398 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.157264    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.157398 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.157331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-tls\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.157398 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.157370    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtccn\" (UniqueName: \"kubernetes.io/projected/624e76da-697c-4deb-8d42-12eecc3ff4a7-kube-api-access-dtccn\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.258144 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.258085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.258144 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.258127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.258285 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.258248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.258332 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.258297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-grpc-tls\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.258364 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.258338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624e76da-697c-4deb-8d42-12eecc3ff4a7-metrics-client-ca\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.258414 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.258385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.258466 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.258424    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-tls\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.258832 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.258576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtccn\" (UniqueName: \"kubernetes.io/projected/624e76da-697c-4deb-8d42-12eecc3ff4a7-kube-api-access-dtccn\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.259174 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.259149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/624e76da-697c-4deb-8d42-12eecc3ff4a7-metrics-client-ca\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.260784 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.260763    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.260878 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.260829    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.261039 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.261002    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.261086 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.261053    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.261230 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.261211    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-grpc-tls\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.261538 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.261522    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/624e76da-697c-4deb-8d42-12eecc3ff4a7-secret-thanos-querier-tls\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.266490 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.266473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtccn\" (UniqueName: \"kubernetes.io/projected/624e76da-697c-4deb-8d42-12eecc3ff4a7-kube-api-access-dtccn\") pod \"thanos-querier-57889dc56-7n8pq\" (UID: \"624e76da-697c-4deb-8d42-12eecc3ff4a7\") " pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.402373 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.402340    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:10.427432 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.427397    2558 generic.go:356] "Generic (PLEG): container finished" podID="99cb1324-7f9a-45a4-9c2b-be4fc936ad44" containerID="099a6c3c71e0270d9e67c2f8b7a32deeb00bf4a09e0f218bfeef027ae5a12ded" exitCode=0
Jul 31 14:17:10.427811 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.427480    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfhj9" event={"ID":"99cb1324-7f9a-45a4-9c2b-be4fc936ad44","Type":"ContainerDied","Data":"099a6c3c71e0270d9e67c2f8b7a32deeb00bf4a09e0f218bfeef027ae5a12ded"}
Jul 31 14:17:10.555999 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:10.555977    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57889dc56-7n8pq"]
Jul 31 14:17:10.558007 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:17:10.557986    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624e76da_697c_4deb_8d42_12eecc3ff4a7.slice/crio-24fbdb303f094578f4cfbbb641484101b569bacf11791431b8e20700248e4c8a WatchSource:0}: Error finding container 24fbdb303f094578f4cfbbb641484101b569bacf11791431b8e20700248e4c8a: Status 404 returned error can't find the container with id 24fbdb303f094578f4cfbbb641484101b569bacf11791431b8e20700248e4c8a
Jul 31 14:17:11.434889 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:11.434851    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfhj9" event={"ID":"99cb1324-7f9a-45a4-9c2b-be4fc936ad44","Type":"ContainerStarted","Data":"0213838da6a05848f7023594e9e6018b57f80100f59c9e6b79fffe0965ea1a65"}
Jul 31 14:17:11.435373 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:11.434918    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kfhj9" event={"ID":"99cb1324-7f9a-45a4-9c2b-be4fc936ad44","Type":"ContainerStarted","Data":"1de7615058ea072b4ef160f20af6e53a4183010a4a59d444be963e4042592911"}
Jul 31 14:17:11.436626 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:11.436561    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq" event={"ID":"624e76da-697c-4deb-8d42-12eecc3ff4a7","Type":"ContainerStarted","Data":"24fbdb303f094578f4cfbbb641484101b569bacf11791431b8e20700248e4c8a"}
Jul 31 14:17:11.457311 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:11.457259    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kfhj9" podStartSLOduration=1.8320544600000002 podStartE2EDuration="3.457238758s" totalImagesPullingTime="1.522152022s" totalInitContainerRuntime="103.032276ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:11.456239283 +0000 UTC m=+270.328802558" watchObservedRunningTime="2026-07-31 14:17:11.457238758 +0000 UTC m=+270.329802013"
Jul 31 14:17:12.573845 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:12.573821    2558 scope.go:117] "RemoveContainer" containerID="c47ab862284df41613f54e0ac0c0b9fba2735d5589ef83028c565dee03288f35"
Jul 31 14:17:12.574206 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:17:12.574033    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-ll4lr_openshift-console-operator(5d416104-70a0-4285-bb1c-5ff965a4f9f4)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" podUID="5d416104-70a0-4285-bb1c-5ff965a4f9f4"
Jul 31 14:17:13.447655 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:13.447622    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq" event={"ID":"624e76da-697c-4deb-8d42-12eecc3ff4a7","Type":"ContainerStarted","Data":"a0470f574c6653677b3b2bc6b350d54badd4e50a586db38131c2d08d4ace5b79"}
Jul 31 14:17:13.447825 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:13.447702    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq" event={"ID":"624e76da-697c-4deb-8d42-12eecc3ff4a7","Type":"ContainerStarted","Data":"ff6aef08c2e8806821420d6d056342eff609d2f3ba7301fadfd632ade16a75a8"}
Jul 31 14:17:13.447825 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:13.447742    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq" event={"ID":"624e76da-697c-4deb-8d42-12eecc3ff4a7","Type":"ContainerStarted","Data":"ad39eeb11846a6a7b9a2cfb0726b926c8689f86abecec2aa6bbe86bf4348b806"}
Jul 31 14:17:14.453866 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:14.453834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq" event={"ID":"624e76da-697c-4deb-8d42-12eecc3ff4a7","Type":"ContainerStarted","Data":"54dd8baab4879aa7ed583a8ad1f80b54f327fc7f247ce157b045695e42c40386"}
Jul 31 14:17:14.454356 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:14.453893    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq" event={"ID":"624e76da-697c-4deb-8d42-12eecc3ff4a7","Type":"ContainerStarted","Data":"49244becbec54be177925132fcce1262f01b3c5f6d43c66f1616aa1a637907f0"}
Jul 31 14:17:14.454356 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:14.453931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq" event={"ID":"624e76da-697c-4deb-8d42-12eecc3ff4a7","Type":"ContainerStarted","Data":"7042e66ca882bca9f289b8edeae845f194a9c0d9206e920aeb7ad1c5c1c7d6ab"}
Jul 31 14:17:14.454356 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:14.454018    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:14.479714 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:14.479667    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq" podStartSLOduration=1.6852569640000001 podStartE2EDuration="4.479652107s" totalImagesPullingTime="2.794395143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:14.478707871 +0000 UTC m=+273.351271150" watchObservedRunningTime="2026-07-31 14:17:14.479652107 +0000 UTC m=+273.352215422"
Jul 31 14:17:18.625129 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:18.625096    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:17:18.973504 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:18.973425    2558 patch_prober.go:28] interesting pod/image-registry-5bcc48f99d-w2dpl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 31 14:17:18.973504 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:18.973477    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl" podUID="6b2cc95b-de81-47cf-a8d2-673a511e70fd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:17:20.391964 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:20.391937    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bcc48f99d-w2dpl"
Jul 31 14:17:20.463313 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:20.463286    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57889dc56-7n8pq"
Jul 31 14:17:22.057899 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:17:22.057855    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fmdw8" podUID="efbfc503-7eb7-435f-94a8-49b4fc9ecdd4"
Jul 31 14:17:22.482664 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:22.482585    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:17:23.639226 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.639180    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-88648cb45-9npzq" podUID="d648bb3e-dc50-4715-b612-08f683abebd8" containerName="registry" containerID="cri-o://0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2" gracePeriod=30
Jul 31 14:17:23.877953 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.877930    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:17:23.978574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.978482    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-installation-pull-secrets\") pod \"d648bb3e-dc50-4715-b612-08f683abebd8\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") "
Jul 31 14:17:23.978574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.978541    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-bound-sa-token\") pod \"d648bb3e-dc50-4715-b612-08f683abebd8\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") "
Jul 31 14:17:23.978574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.978559    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-registry-certificates\") pod \"d648bb3e-dc50-4715-b612-08f683abebd8\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") "
Jul 31 14:17:23.978574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.978578    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dctcj\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-kube-api-access-dctcj\") pod \"d648bb3e-dc50-4715-b612-08f683abebd8\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") "
Jul 31 14:17:23.978963 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.978636    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-trusted-ca\") pod \"d648bb3e-dc50-4715-b612-08f683abebd8\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") "
Jul 31 14:17:23.978963 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.978659    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-image-registry-private-configuration\") pod \"d648bb3e-dc50-4715-b612-08f683abebd8\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") "
Jul 31 14:17:23.978963 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.978675    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") pod \"d648bb3e-dc50-4715-b612-08f683abebd8\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") "
Jul 31 14:17:23.978963 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.978707    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d648bb3e-dc50-4715-b612-08f683abebd8-ca-trust-extracted\") pod \"d648bb3e-dc50-4715-b612-08f683abebd8\" (UID: \"d648bb3e-dc50-4715-b612-08f683abebd8\") "
Jul 31 14:17:23.979206 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.979178    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d648bb3e-dc50-4715-b612-08f683abebd8" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:17:23.979278 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.979257    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d648bb3e-dc50-4715-b612-08f683abebd8" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:17:23.980954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.980923    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d648bb3e-dc50-4715-b612-08f683abebd8" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:17:23.981093 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.980987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-kube-api-access-dctcj" (OuterVolumeSpecName: "kube-api-access-dctcj") pod "d648bb3e-dc50-4715-b612-08f683abebd8" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8"). InnerVolumeSpecName "kube-api-access-dctcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:17:23.981272 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.981249    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d648bb3e-dc50-4715-b612-08f683abebd8" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:17:23.981334 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.981264    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d648bb3e-dc50-4715-b612-08f683abebd8" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:17:23.981334 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.981296    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d648bb3e-dc50-4715-b612-08f683abebd8" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:17:23.987430 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:23.987409    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d648bb3e-dc50-4715-b612-08f683abebd8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d648bb3e-dc50-4715-b612-08f683abebd8" (UID: "d648bb3e-dc50-4715-b612-08f683abebd8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:17:24.079498 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.079466    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-image-registry-private-configuration\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:17:24.079498 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.079492    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-registry-tls\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:17:24.079498 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.079503    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d648bb3e-dc50-4715-b612-08f683abebd8-ca-trust-extracted\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:17:24.079697 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.079512    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d648bb3e-dc50-4715-b612-08f683abebd8-installation-pull-secrets\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:17:24.079697 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.079521    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-bound-sa-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:17:24.079697 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.079529    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-registry-certificates\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:17:24.079697 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.079537    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dctcj\" (UniqueName: \"kubernetes.io/projected/d648bb3e-dc50-4715-b612-08f683abebd8-kube-api-access-dctcj\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:17:24.079697 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.079545    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d648bb3e-dc50-4715-b612-08f683abebd8-trusted-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:17:24.490377 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.490341    2558 generic.go:356] "Generic (PLEG): container finished" podID="d648bb3e-dc50-4715-b612-08f683abebd8" containerID="0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2" exitCode=0
Jul 31 14:17:24.490522 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.490386    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-88648cb45-9npzq"
Jul 31 14:17:24.490522 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.490434    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-88648cb45-9npzq" event={"ID":"d648bb3e-dc50-4715-b612-08f683abebd8","Type":"ContainerDied","Data":"0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2"}
Jul 31 14:17:24.490522 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.490484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-88648cb45-9npzq" event={"ID":"d648bb3e-dc50-4715-b612-08f683abebd8","Type":"ContainerDied","Data":"93c0f00e26f2b722c039a28d571e4f1e665f32a97e8d5b8e1606818088e36c2e"}
Jul 31 14:17:24.490522 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.490507    2558 scope.go:117] "RemoveContainer" containerID="0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2"
Jul 31 14:17:24.499184 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.499167    2558 scope.go:117] "RemoveContainer" containerID="0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2"
Jul 31 14:17:24.499462 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:17:24.499439    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2\": container with ID starting with 0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2 not found: ID does not exist" containerID="0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2"
Jul 31 14:17:24.499512 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.499470    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2"} err="failed to get container status \"0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2\": rpc error: code = NotFound desc = could not find container \"0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2\": container with ID starting with 0d0c995c3b93907aa2f643650d592df97655f7f1ab69c3fdbfcc16df220939e2 not found: ID does not exist"
Jul 31 14:17:24.511851 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.511824    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-88648cb45-9npzq"]
Jul 31 14:17:24.517850 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:24.517831    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-88648cb45-9npzq"]
Jul 31 14:17:25.577802 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:25.577764    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d648bb3e-dc50-4715-b612-08f683abebd8" path="/var/lib/kubelet/pods/d648bb3e-dc50-4715-b612-08f683abebd8/volumes"
Jul 31 14:17:25.595184 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:25.595155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:17:25.597449 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:25.597431    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/efbfc503-7eb7-435f-94a8-49b4fc9ecdd4-metrics-tls\") pod \"dns-default-fmdw8\" (UID: \"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4\") " pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:17:25.786420 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:25.786388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p9vdf\""
Jul 31 14:17:25.794452 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:25.794432    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:17:25.920581 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:25.920501    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fmdw8"]
Jul 31 14:17:25.922713 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:17:25.922681    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefbfc503_7eb7_435f_94a8_49b4fc9ecdd4.slice/crio-5c2cfd6c71f1a2fcace2a8feca8e25d1bc77b08fe32f5fc4e9e3d445b2eaa352 WatchSource:0}: Error finding container 5c2cfd6c71f1a2fcace2a8feca8e25d1bc77b08fe32f5fc4e9e3d445b2eaa352: Status 404 returned error can't find the container with id 5c2cfd6c71f1a2fcace2a8feca8e25d1bc77b08fe32f5fc4e9e3d445b2eaa352
Jul 31 14:17:26.501236 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:26.501196    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fmdw8" event={"ID":"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4","Type":"ContainerStarted","Data":"5c2cfd6c71f1a2fcace2a8feca8e25d1bc77b08fe32f5fc4e9e3d445b2eaa352"}
Jul 31 14:17:27.574261 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:27.574233    2558 scope.go:117] "RemoveContainer" containerID="c47ab862284df41613f54e0ac0c0b9fba2735d5589ef83028c565dee03288f35"
Jul 31 14:17:28.513188 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.513156    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:17:28.513360 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.513288    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" event={"ID":"5d416104-70a0-4285-bb1c-5ff965a4f9f4","Type":"ContainerStarted","Data":"665bd0f1ef5208b13040b682517b45228aaf0f2bafbbd22d8f8b387686cdeb4b"}
Jul 31 14:17:28.513708 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.513619    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:17:28.516309 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.515723    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fmdw8" event={"ID":"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4","Type":"ContainerStarted","Data":"ffa439c1a509ddbd94113efa6d9ea869117402f03d9f351cfbd2ea684e26ab9b"}
Jul 31 14:17:28.516309 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.515774    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fmdw8" event={"ID":"efbfc503-7eb7-435f-94a8-49b4fc9ecdd4","Type":"ContainerStarted","Data":"71d4ad7ecd9f6492a47c68d16ce718b58a8fd822d6db1161d7801791a1555976"}
Jul 31 14:17:28.516504 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.516312    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:17:28.520117 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.520098    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr"
Jul 31 14:17:28.536191 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.536134    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-ll4lr" podStartSLOduration=56.864669008999996 podStartE2EDuration="59.536114307s" totalImagesPullingTime="2.671445298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:28.534655321 +0000 UTC m=+287.407218608" watchObservedRunningTime="2026-07-31 14:17:28.536114307 +0000 UTC m=+287.408677561"
Jul 31 14:17:28.554367 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.554320    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fmdw8" podStartSLOduration=252.022242086 podStartE2EDuration="4m13.554305037s" totalImagesPullingTime="1.532062951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:28.553410067 +0000 UTC m=+287.425973324" watchObservedRunningTime="2026-07-31 14:17:28.554305037 +0000 UTC m=+287.426868328"
Jul 31 14:17:28.563528 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.563504    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-c6s6l"]
Jul 31 14:17:28.563794 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.563782    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d648bb3e-dc50-4715-b612-08f683abebd8" containerName="registry"
Jul 31 14:17:28.563839 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.563796    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d648bb3e-dc50-4715-b612-08f683abebd8" containerName="registry"
Jul 31 14:17:28.563873 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.563857    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d648bb3e-dc50-4715-b612-08f683abebd8" containerName="registry"
Jul 31 14:17:28.565784 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.565761    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-c6s6l"
Jul 31 14:17:28.568351 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.568329    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-m6h4j\""
Jul 31 14:17:28.568351 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.568329    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 14:17:28.568511 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.568333    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 14:17:28.576146 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.576124    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-c6s6l"]
Jul 31 14:17:28.626146 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.626114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2ms\" (UniqueName: \"kubernetes.io/projected/8af6e796-b111-4aa2-91c9-77a2c7e7b6c0-kube-api-access-6d2ms\") pod \"downloads-77c974ffcf-c6s6l\" (UID: \"8af6e796-b111-4aa2-91c9-77a2c7e7b6c0\") " pod="openshift-console/downloads-77c974ffcf-c6s6l"
Jul 31 14:17:28.727251 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.727214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2ms\" (UniqueName: \"kubernetes.io/projected/8af6e796-b111-4aa2-91c9-77a2c7e7b6c0-kube-api-access-6d2ms\") pod \"downloads-77c974ffcf-c6s6l\" (UID: \"8af6e796-b111-4aa2-91c9-77a2c7e7b6c0\") " pod="openshift-console/downloads-77c974ffcf-c6s6l"
Jul 31 14:17:28.735448 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.735425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2ms\" (UniqueName: \"kubernetes.io/projected/8af6e796-b111-4aa2-91c9-77a2c7e7b6c0-kube-api-access-6d2ms\") pod \"downloads-77c974ffcf-c6s6l\" (UID: \"8af6e796-b111-4aa2-91c9-77a2c7e7b6c0\") " pod="openshift-console/downloads-77c974ffcf-c6s6l"
Jul 31 14:17:28.875690 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:28.875660    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-c6s6l"
Jul 31 14:17:29.002008 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:29.001982    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-c6s6l"]
Jul 31 14:17:29.004574 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:17:29.004542    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af6e796_b111_4aa2_91c9_77a2c7e7b6c0.slice/crio-96df00683c2433283a514a413e1b4af71b99ad01334108dafba02d86bbb82088 WatchSource:0}: Error finding container 96df00683c2433283a514a413e1b4af71b99ad01334108dafba02d86bbb82088: Status 404 returned error can't find the container with id 96df00683c2433283a514a413e1b4af71b99ad01334108dafba02d86bbb82088
Jul 31 14:17:29.520954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:29.520918    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-c6s6l" event={"ID":"8af6e796-b111-4aa2-91c9-77a2c7e7b6c0","Type":"ContainerStarted","Data":"96df00683c2433283a514a413e1b4af71b99ad01334108dafba02d86bbb82088"}
Jul 31 14:17:38.999864 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:38.999830    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64c6c9f48d-4jdvk"]
Jul 31 14:17:39.002565 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.002544    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.005252 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.005229    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 31 14:17:39.005385 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.005274    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 31 14:17:39.006389 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.006356    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 31 14:17:39.006389 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.006379    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 31 14:17:39.006581 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.006366    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 31 14:17:39.006707 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.006681    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bv27n\""
Jul 31 14:17:39.011922 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.011897    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c6c9f48d-4jdvk"]
Jul 31 14:17:39.124278 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.124236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjc4k\" (UniqueName: \"kubernetes.io/projected/2f8d56ce-2014-42f5-866d-66e711f4bd98-kube-api-access-cjc4k\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.124278 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.124283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-serving-cert\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.124505 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.124355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-config\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.124505 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.124391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-service-ca\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.124505 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.124435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-oauth-config\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.124505 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.124455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-oauth-serving-cert\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.224989 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.224955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-oauth-serving-cert\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.225254 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.225001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjc4k\" (UniqueName: \"kubernetes.io/projected/2f8d56ce-2014-42f5-866d-66e711f4bd98-kube-api-access-cjc4k\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.225254 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.225054    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-serving-cert\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.225254 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.225088    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-config\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.225254 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.225127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-service-ca\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.225254 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.225181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-oauth-config\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.225960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.225931    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-config\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.226112 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.225973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-oauth-serving-cert\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.226112 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.226042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-service-ca\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.228147 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.228122    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-oauth-config\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.228278 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.228262    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-serving-cert\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.233279 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.233261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjc4k\" (UniqueName: \"kubernetes.io/projected/2f8d56ce-2014-42f5-866d-66e711f4bd98-kube-api-access-cjc4k\") pod \"console-64c6c9f48d-4jdvk\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") " pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.314955 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.314921    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:39.529248 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:39.529217    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fmdw8"
Jul 31 14:17:45.277224 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.277194    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:17:45.279810 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.279777    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:17:45.293893 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.293868    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 14:17:45.372434 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.372410    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64c6c9f48d-4jdvk"]
Jul 31 14:17:45.374985 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:17:45.374959    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8d56ce_2014_42f5_866d_66e711f4bd98.slice/crio-373aa6776e3c03c4939455694935cd39dd9f81f16a739f4b81399dbf9dce067e WatchSource:0}: Error finding container 373aa6776e3c03c4939455694935cd39dd9f81f16a739f4b81399dbf9dce067e: Status 404 returned error can't find the container with id 373aa6776e3c03c4939455694935cd39dd9f81f16a739f4b81399dbf9dce067e
Jul 31 14:17:45.377554 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.377533    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:17:45.601235 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.601195    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c6c9f48d-4jdvk" event={"ID":"2f8d56ce-2014-42f5-866d-66e711f4bd98","Type":"ContainerStarted","Data":"373aa6776e3c03c4939455694935cd39dd9f81f16a739f4b81399dbf9dce067e"}
Jul 31 14:17:45.603070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.603008    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-c6s6l" event={"ID":"8af6e796-b111-4aa2-91c9-77a2c7e7b6c0","Type":"ContainerStarted","Data":"3f638b27b3d5b23ff87fa07cc15e2507e04637f733fd6be052fac650c14c577d"}
Jul 31 14:17:45.603249 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.603221    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-c6s6l"
Jul 31 14:17:45.620299 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.620253    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-c6s6l" podStartSLOduration=1.309098625 podStartE2EDuration="17.620231639s" totalImagesPullingTime="16.311133014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:45.619191224 +0000 UTC m=+304.491754468" watchObservedRunningTime="2026-07-31 14:17:45.620231639 +0000 UTC m=+304.492794893"
Jul 31 14:17:45.623942 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:45.623911    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-c6s6l"
Jul 31 14:17:49.624981 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:49.624940    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c6c9f48d-4jdvk" event={"ID":"2f8d56ce-2014-42f5-866d-66e711f4bd98","Type":"ContainerStarted","Data":"cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a"}
Jul 31 14:17:49.644702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:49.644646    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64c6c9f48d-4jdvk" podStartSLOduration=7.872654735 podStartE2EDuration="11.644628522s" totalImagesPullingTime="3.771973787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:17:49.642196366 +0000 UTC m=+308.514759632" watchObservedRunningTime="2026-07-31 14:17:49.644628522 +0000 UTC m=+308.517191774"
Jul 31 14:17:55.651394 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:55.651356    2558 generic.go:356] "Generic (PLEG): container finished" podID="6aeed834-fbee-468e-ab99-ac7121d9fb08" containerID="45fe8200a2886398d3eac544334a217eab86260d395c6a846a36e5376f185e6f" exitCode=0
Jul 31 14:17:55.651804 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:55.651412    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z" event={"ID":"6aeed834-fbee-468e-ab99-ac7121d9fb08","Type":"ContainerDied","Data":"45fe8200a2886398d3eac544334a217eab86260d395c6a846a36e5376f185e6f"}
Jul 31 14:17:55.651804 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:55.651743    2558 scope.go:117] "RemoveContainer" containerID="45fe8200a2886398d3eac544334a217eab86260d395c6a846a36e5376f185e6f"
Jul 31 14:17:56.656893 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:56.656856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-pd68z" event={"ID":"6aeed834-fbee-468e-ab99-ac7121d9fb08","Type":"ContainerStarted","Data":"30c95d07abc7eb141caba89865083304c81ef2841578cdb7b89f0f3093a6c140"}
Jul 31 14:17:59.315458 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:59.315422    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:59.315866 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:59.315494    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:59.320361 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:59.320340    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:17:59.671912 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:17:59.671837    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:18:00.076573 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:00.076536    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c6c9f48d-4jdvk"]
Jul 31 14:18:05.691677 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:05.691642    2558 generic.go:356] "Generic (PLEG): container finished" podID="c6e38f19-a6c6-4eb8-909b-df5296e6655e" containerID="a6ff87080eeb0b3e6da55dfe871fa59f5bf8f8f1de7262f71141fe1fe800c234" exitCode=0
Jul 31 14:18:05.692163 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:05.691714    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-lb47s" event={"ID":"c6e38f19-a6c6-4eb8-909b-df5296e6655e","Type":"ContainerDied","Data":"a6ff87080eeb0b3e6da55dfe871fa59f5bf8f8f1de7262f71141fe1fe800c234"}
Jul 31 14:18:05.692163 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:05.692077    2558 scope.go:117] "RemoveContainer" containerID="a6ff87080eeb0b3e6da55dfe871fa59f5bf8f8f1de7262f71141fe1fe800c234"
Jul 31 14:18:06.697149 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:06.697113    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-lb47s" event={"ID":"c6e38f19-a6c6-4eb8-909b-df5296e6655e","Type":"ContainerStarted","Data":"cf981f1d62d3909ec76c1a6dc42856d7f1e3014f1321cd4de0c795e44231af86"}
Jul 31 14:18:08.706087 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:08.705994    2558 generic.go:356] "Generic (PLEG): container finished" podID="7355b00f-f3fb-4871-83ee-97769c6c2e6b" containerID="9a5be18b0b7be7343690151cc588f06ffa23416461719da0f4eb8c388b655bdc" exitCode=0
Jul 31 14:18:08.706087 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:08.706070    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg" event={"ID":"7355b00f-f3fb-4871-83ee-97769c6c2e6b","Type":"ContainerDied","Data":"9a5be18b0b7be7343690151cc588f06ffa23416461719da0f4eb8c388b655bdc"}
Jul 31 14:18:08.706475 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:08.706413    2558 scope.go:117] "RemoveContainer" containerID="9a5be18b0b7be7343690151cc588f06ffa23416461719da0f4eb8c388b655bdc"
Jul 31 14:18:09.712123 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:09.712089    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-5kzfg" event={"ID":"7355b00f-f3fb-4871-83ee-97769c6c2e6b","Type":"ContainerStarted","Data":"b75f264a04e93c7856670b2787ecdd5621e99b2dcef636337b86bd4dedc2eb1e"}
Jul 31 14:18:26.694172 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:26.694123    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64c6c9f48d-4jdvk" podUID="2f8d56ce-2014-42f5-866d-66e711f4bd98" containerName="console" containerID="cri-o://cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a" gracePeriod=15
Jul 31 14:18:26.964104 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:26.964041    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:18:27.023911 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.023877    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-service-ca\") pod \"2f8d56ce-2014-42f5-866d-66e711f4bd98\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") "
Jul 31 14:18:27.024100 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.023930    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cjc4k\" (UniqueName: \"kubernetes.io/projected/2f8d56ce-2014-42f5-866d-66e711f4bd98-kube-api-access-cjc4k\") pod \"2f8d56ce-2014-42f5-866d-66e711f4bd98\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") "
Jul 31 14:18:27.024100 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.023989    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-oauth-config\") pod \"2f8d56ce-2014-42f5-866d-66e711f4bd98\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") "
Jul 31 14:18:27.024100 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.024016    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-oauth-serving-cert\") pod \"2f8d56ce-2014-42f5-866d-66e711f4bd98\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") "
Jul 31 14:18:27.024100 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.024078    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-serving-cert\") pod \"2f8d56ce-2014-42f5-866d-66e711f4bd98\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") "
Jul 31 14:18:27.024302 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.024134    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-config\") pod \"2f8d56ce-2014-42f5-866d-66e711f4bd98\" (UID: \"2f8d56ce-2014-42f5-866d-66e711f4bd98\") "
Jul 31 14:18:27.024350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.024295    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f8d56ce-2014-42f5-866d-66e711f4bd98" (UID: "2f8d56ce-2014-42f5-866d-66e711f4bd98"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:18:27.024399 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.024379    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:18:27.024472 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.024443    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f8d56ce-2014-42f5-866d-66e711f4bd98" (UID: "2f8d56ce-2014-42f5-866d-66e711f4bd98"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:18:27.024549 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.024531    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-config" (OuterVolumeSpecName: "console-config") pod "2f8d56ce-2014-42f5-866d-66e711f4bd98" (UID: "2f8d56ce-2014-42f5-866d-66e711f4bd98"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:18:27.026285 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.026255    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f8d56ce-2014-42f5-866d-66e711f4bd98" (UID: "2f8d56ce-2014-42f5-866d-66e711f4bd98"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:18:27.026381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.026293    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f8d56ce-2014-42f5-866d-66e711f4bd98" (UID: "2f8d56ce-2014-42f5-866d-66e711f4bd98"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:18:27.026381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.026331    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8d56ce-2014-42f5-866d-66e711f4bd98-kube-api-access-cjc4k" (OuterVolumeSpecName: "kube-api-access-cjc4k") pod "2f8d56ce-2014-42f5-866d-66e711f4bd98" (UID: "2f8d56ce-2014-42f5-866d-66e711f4bd98"). InnerVolumeSpecName "kube-api-access-cjc4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:18:27.125141 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.125108    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjc4k\" (UniqueName: \"kubernetes.io/projected/2f8d56ce-2014-42f5-866d-66e711f4bd98-kube-api-access-cjc4k\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:18:27.125141 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.125134    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-oauth-config\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:18:27.125141 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.125143    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-oauth-serving-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:18:27.125141 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.125151    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-serving-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:18:27.125382 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.125160    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f8d56ce-2014-42f5-866d-66e711f4bd98-console-config\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:18:27.791839 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.791801    2558 generic.go:356] "Generic (PLEG): container finished" podID="2f8d56ce-2014-42f5-866d-66e711f4bd98" containerID="cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a" exitCode=0
Jul 31 14:18:27.792345 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.791866    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64c6c9f48d-4jdvk"
Jul 31 14:18:27.792345 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.791875    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c6c9f48d-4jdvk" event={"ID":"2f8d56ce-2014-42f5-866d-66e711f4bd98","Type":"ContainerDied","Data":"cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a"}
Jul 31 14:18:27.792345 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.791941    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64c6c9f48d-4jdvk" event={"ID":"2f8d56ce-2014-42f5-866d-66e711f4bd98","Type":"ContainerDied","Data":"373aa6776e3c03c4939455694935cd39dd9f81f16a739f4b81399dbf9dce067e"}
Jul 31 14:18:27.792345 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.791964    2558 scope.go:117] "RemoveContainer" containerID="cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a"
Jul 31 14:18:27.800190 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.800175    2558 scope.go:117] "RemoveContainer" containerID="cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a"
Jul 31 14:18:27.800449 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:18:27.800428    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a\": container with ID starting with cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a not found: ID does not exist" containerID="cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a"
Jul 31 14:18:27.800556 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.800454    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a"} err="failed to get container status \"cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a\": rpc error: code = NotFound desc = could not find container \"cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a\": container with ID starting with cc9a5f567fc7e912f528a962b5095455588acbd6affe65c622eeb0d9e4f7347a not found: ID does not exist"
Jul 31 14:18:27.808291 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.808269    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64c6c9f48d-4jdvk"]
Jul 31 14:18:27.813638 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:27.813619    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64c6c9f48d-4jdvk"]
Jul 31 14:18:29.579397 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:18:29.579369    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8d56ce-2014-42f5-866d-66e711f4bd98" path="/var/lib/kubelet/pods/2f8d56ce-2014-42f5-866d-66e711f4bd98/volumes"
Jul 31 14:19:56.496744 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.496707    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-688979447f-bb69d"]
Jul 31 14:19:56.497230 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.497043    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8d56ce-2014-42f5-866d-66e711f4bd98" containerName="console"
Jul 31 14:19:56.497230 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.497062    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8d56ce-2014-42f5-866d-66e711f4bd98" containerName="console"
Jul 31 14:19:56.497230 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.497146    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8d56ce-2014-42f5-866d-66e711f4bd98" containerName="console"
Jul 31 14:19:56.499483 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.499457    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.502437 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.502393    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 31 14:19:56.503511 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.503483    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 31 14:19:56.503685 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.503527    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bv27n\""
Jul 31 14:19:56.503685 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.503490    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 31 14:19:56.503685 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.503617    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 31 14:19:56.503685 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.503647    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 31 14:19:56.508608 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.508589    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 31 14:19:56.511619 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.511598    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-688979447f-bb69d"]
Jul 31 14:19:56.621325 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.621289    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l7ff\" (UniqueName: \"kubernetes.io/projected/e29c3784-070c-4687-8206-fab5d3eb778f-kube-api-access-9l7ff\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.621325 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.621327    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-trusted-ca-bundle\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.621538 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.621360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-service-ca\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.621538 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.621410    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-oauth-config\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.621538 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.621436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-oauth-serving-cert\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.621538 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.621516    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-serving-cert\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.621678 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.621543    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-console-config\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.722642 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.722602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-trusted-ca-bundle\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.722837 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.722655    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-service-ca\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.722837 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.722683    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-oauth-config\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.722837 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.722707    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-oauth-serving-cert\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.722837 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.722753    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-serving-cert\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.722837 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.722779    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-console-config\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.723108 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.722853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l7ff\" (UniqueName: \"kubernetes.io/projected/e29c3784-070c-4687-8206-fab5d3eb778f-kube-api-access-9l7ff\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.723565 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.723535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-oauth-serving-cert\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.723690 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.723535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-service-ca\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.723690 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.723547    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-console-config\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.723813 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.723789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-trusted-ca-bundle\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.725738 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.725713    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-oauth-config\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.725824 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.725803    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-serving-cert\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.730454 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.730432    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l7ff\" (UniqueName: \"kubernetes.io/projected/e29c3784-070c-4687-8206-fab5d3eb778f-kube-api-access-9l7ff\") pod \"console-688979447f-bb69d\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") " pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.809769 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.809741    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:19:56.947417 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:56.947390    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-688979447f-bb69d"]
Jul 31 14:19:56.947532 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:19:56.947388    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29c3784_070c_4687_8206_fab5d3eb778f.slice/crio-ef6708c0d85dd01366462465eaa8995b3105ae4ac249d93a14fb24d40534ce7e WatchSource:0}: Error finding container ef6708c0d85dd01366462465eaa8995b3105ae4ac249d93a14fb24d40534ce7e: Status 404 returned error can't find the container with id ef6708c0d85dd01366462465eaa8995b3105ae4ac249d93a14fb24d40534ce7e
Jul 31 14:19:57.125103 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:57.125002    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688979447f-bb69d" event={"ID":"e29c3784-070c-4687-8206-fab5d3eb778f","Type":"ContainerStarted","Data":"907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878"}
Jul 31 14:19:57.125103 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:57.125074    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688979447f-bb69d" event={"ID":"e29c3784-070c-4687-8206-fab5d3eb778f","Type":"ContainerStarted","Data":"ef6708c0d85dd01366462465eaa8995b3105ae4ac249d93a14fb24d40534ce7e"}
Jul 31 14:19:57.141781 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:19:57.141727    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-688979447f-bb69d" podStartSLOduration=1.141707781 podStartE2EDuration="1.141707781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:19:57.140847157 +0000 UTC m=+436.013410409" watchObservedRunningTime="2026-07-31 14:19:57.141707781 +0000 UTC m=+436.014271036"
Jul 31 14:20:06.810403 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:20:06.810309    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:20:06.810403 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:20:06.810385    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:20:06.815169 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:20:06.815147    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:20:07.166839 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:20:07.166753    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:22:00.732206 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.732169    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"]
Jul 31 14:22:00.734446 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.734430    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"
Jul 31 14:22:00.737009 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.736980    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:22:00.737138 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.737005    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 31 14:22:00.737138 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.737079    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-m46lh\""
Jul 31 14:22:00.747586 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.745927    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"]
Jul 31 14:22:00.832461 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.832426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f808120d-18db-45ca-8328-c3f26f7ac237-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjd78\" (UID: \"f808120d-18db-45ca-8328-c3f26f7ac237\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"
Jul 31 14:22:00.832651 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.832481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xxs\" (UniqueName: \"kubernetes.io/projected/f808120d-18db-45ca-8328-c3f26f7ac237-kube-api-access-r5xxs\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjd78\" (UID: \"f808120d-18db-45ca-8328-c3f26f7ac237\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"
Jul 31 14:22:00.933464 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.933421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xxs\" (UniqueName: \"kubernetes.io/projected/f808120d-18db-45ca-8328-c3f26f7ac237-kube-api-access-r5xxs\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjd78\" (UID: \"f808120d-18db-45ca-8328-c3f26f7ac237\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"
Jul 31 14:22:00.933661 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.933528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f808120d-18db-45ca-8328-c3f26f7ac237-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjd78\" (UID: \"f808120d-18db-45ca-8328-c3f26f7ac237\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"
Jul 31 14:22:00.933912 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.933895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f808120d-18db-45ca-8328-c3f26f7ac237-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjd78\" (UID: \"f808120d-18db-45ca-8328-c3f26f7ac237\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"
Jul 31 14:22:00.948869 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:00.948838    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xxs\" (UniqueName: \"kubernetes.io/projected/f808120d-18db-45ca-8328-c3f26f7ac237-kube-api-access-r5xxs\") pod \"cert-manager-operator-controller-manager-55c444bf86-cjd78\" (UID: \"f808120d-18db-45ca-8328-c3f26f7ac237\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"
Jul 31 14:22:01.050057 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:01.050031    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"
Jul 31 14:22:01.201883 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:01.201859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78"]
Jul 31 14:22:01.204414 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:22:01.204380    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf808120d_18db_45ca_8328_c3f26f7ac237.slice/crio-f16f33d4826baf0df4773d6eb2a90d5ab283ffa3fb196a72d4318c0dbfe6dffa WatchSource:0}: Error finding container f16f33d4826baf0df4773d6eb2a90d5ab283ffa3fb196a72d4318c0dbfe6dffa: Status 404 returned error can't find the container with id f16f33d4826baf0df4773d6eb2a90d5ab283ffa3fb196a72d4318c0dbfe6dffa
Jul 31 14:22:01.596602 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:01.596565    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78" event={"ID":"f808120d-18db-45ca-8328-c3f26f7ac237","Type":"ContainerStarted","Data":"f16f33d4826baf0df4773d6eb2a90d5ab283ffa3fb196a72d4318c0dbfe6dffa"}
Jul 31 14:22:11.384350 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:22:11.384304    2558 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:d8a14cf03a7872a88dd61e1aa6b2a84ce031a2044d0d4d129154041a1adaa481: Requesting bearer token: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:d8a14cf03a7872a88dd61e1aa6b2a84ce031a2044d0d4d129154041a1adaa481"
Jul 31 14:22:11.384788 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:22:11.384435    2558 kuberuntime_manager.go:1459] "Unhandled Error" err="container cert-manager-operator start failed in pod cert-manager-operator-controller-manager-55c444bf86-cjd78_cert-manager-operator(f808120d-18db-45ca-8328-c3f26f7ac237): ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:d8a14cf03a7872a88dd61e1aa6b2a84ce031a2044d0d4d129154041a1adaa481: Requesting bearer token: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 31 14:22:11.384788 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:22:11.384490    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-operator\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:d8a14cf03a7872a88dd61e1aa6b2a84ce031a2044d0d4d129154041a1adaa481: Requesting bearer token: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78" podUID="f808120d-18db-45ca-8328-c3f26f7ac237"
Jul 31 14:22:11.638179 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:22:11.638148    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-operator\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:d8a14cf03a7872a88dd61e1aa6b2a84ce031a2044d0d4d129154041a1adaa481\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:d8a14cf03a7872a88dd61e1aa6b2a84ce031a2044d0d4d129154041a1adaa481: Requesting bearer token: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78" podUID="f808120d-18db-45ca-8328-c3f26f7ac237"
Jul 31 14:22:29.719540 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:29.719499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78" event={"ID":"f808120d-18db-45ca-8328-c3f26f7ac237","Type":"ContainerStarted","Data":"ce41445b83e8985322662e173a617b40f87659b8b3481fe8423330ef7b7d4770"}
Jul 31 14:22:29.745667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:29.745608    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cjd78" podStartSLOduration=1.8495144890000002 podStartE2EDuration="29.745586597s" totalImagesPullingTime="27.896072108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-31 14:22:29.742666405 +0000 UTC m=+588.615229697" watchObservedRunningTime="2026-07-31 14:22:29.745586597 +0000 UTC m=+588.618149857"
Jul 31 14:22:42.287489 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.287450    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mrt97"]
Jul 31 14:22:42.289507 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.289492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mrt97"
Jul 31 14:22:42.291936 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.291915    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 31 14:22:42.291936 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.291915    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-657qm\""
Jul 31 14:22:42.292872 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.292857    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 31 14:22:42.298510 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.298489    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mrt97"]
Jul 31 14:22:42.467916 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.467880    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a09a135-e020-4174-87ff-ea7d49dbbb36-bound-sa-token\") pod \"cert-manager-55b4d954d5-mrt97\" (UID: \"1a09a135-e020-4174-87ff-ea7d49dbbb36\") " pod="cert-manager/cert-manager-55b4d954d5-mrt97"
Jul 31 14:22:42.468107 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.467931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrjh\" (UniqueName: \"kubernetes.io/projected/1a09a135-e020-4174-87ff-ea7d49dbbb36-kube-api-access-tkrjh\") pod \"cert-manager-55b4d954d5-mrt97\" (UID: \"1a09a135-e020-4174-87ff-ea7d49dbbb36\") " pod="cert-manager/cert-manager-55b4d954d5-mrt97"
Jul 31 14:22:42.568826 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.568798    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a09a135-e020-4174-87ff-ea7d49dbbb36-bound-sa-token\") pod \"cert-manager-55b4d954d5-mrt97\" (UID: \"1a09a135-e020-4174-87ff-ea7d49dbbb36\") " pod="cert-manager/cert-manager-55b4d954d5-mrt97"
Jul 31 14:22:42.569007 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.568843    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrjh\" (UniqueName: \"kubernetes.io/projected/1a09a135-e020-4174-87ff-ea7d49dbbb36-kube-api-access-tkrjh\") pod \"cert-manager-55b4d954d5-mrt97\" (UID: \"1a09a135-e020-4174-87ff-ea7d49dbbb36\") " pod="cert-manager/cert-manager-55b4d954d5-mrt97"
Jul 31 14:22:42.576754 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.576724    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a09a135-e020-4174-87ff-ea7d49dbbb36-bound-sa-token\") pod \"cert-manager-55b4d954d5-mrt97\" (UID: \"1a09a135-e020-4174-87ff-ea7d49dbbb36\") " pod="cert-manager/cert-manager-55b4d954d5-mrt97"
Jul 31 14:22:42.576850 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.576836    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrjh\" (UniqueName: \"kubernetes.io/projected/1a09a135-e020-4174-87ff-ea7d49dbbb36-kube-api-access-tkrjh\") pod \"cert-manager-55b4d954d5-mrt97\" (UID: \"1a09a135-e020-4174-87ff-ea7d49dbbb36\") " pod="cert-manager/cert-manager-55b4d954d5-mrt97"
Jul 31 14:22:42.615700 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.615675    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mrt97"
Jul 31 14:22:42.742645 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.742622    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mrt97"]
Jul 31 14:22:42.745053 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:22:42.745005    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a09a135_e020_4174_87ff_ea7d49dbbb36.slice/crio-25492509860ad532816c065006599fdbf94d8a671869e1b3d0f506624d969007 WatchSource:0}: Error finding container 25492509860ad532816c065006599fdbf94d8a671869e1b3d0f506624d969007: Status 404 returned error can't find the container with id 25492509860ad532816c065006599fdbf94d8a671869e1b3d0f506624d969007
Jul 31 14:22:42.777217 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:42.777189    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mrt97" event={"ID":"1a09a135-e020-4174-87ff-ea7d49dbbb36","Type":"ContainerStarted","Data":"25492509860ad532816c065006599fdbf94d8a671869e1b3d0f506624d969007"}
Jul 31 14:22:44.203761 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.203721    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"]
Jul 31 14:22:44.206458 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.206437    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"
Jul 31 14:22:44.209061 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.209039    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 31 14:22:44.209195 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.209067    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fgrgd\""
Jul 31 14:22:44.209935 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.209918    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:22:44.214512 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.214487    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"]
Jul 31 14:22:44.383018 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.382982    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec566262-22c4-4de3-826e-fcc0c7777ef0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b58z8\" (UID: \"ec566262-22c4-4de3-826e-fcc0c7777ef0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"
Jul 31 14:22:44.383173 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.383098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82lcl\" (UniqueName: \"kubernetes.io/projected/ec566262-22c4-4de3-826e-fcc0c7777ef0-kube-api-access-82lcl\") pod \"openshift-lws-operator-fd8ccff4c-b58z8\" (UID: \"ec566262-22c4-4de3-826e-fcc0c7777ef0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"
Jul 31 14:22:44.483574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.483490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82lcl\" (UniqueName: \"kubernetes.io/projected/ec566262-22c4-4de3-826e-fcc0c7777ef0-kube-api-access-82lcl\") pod \"openshift-lws-operator-fd8ccff4c-b58z8\" (UID: \"ec566262-22c4-4de3-826e-fcc0c7777ef0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"
Jul 31 14:22:44.483574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.483566    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec566262-22c4-4de3-826e-fcc0c7777ef0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b58z8\" (UID: \"ec566262-22c4-4de3-826e-fcc0c7777ef0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"
Jul 31 14:22:44.483981 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.483961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec566262-22c4-4de3-826e-fcc0c7777ef0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-b58z8\" (UID: \"ec566262-22c4-4de3-826e-fcc0c7777ef0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"
Jul 31 14:22:44.493110 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.493086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82lcl\" (UniqueName: \"kubernetes.io/projected/ec566262-22c4-4de3-826e-fcc0c7777ef0-kube-api-access-82lcl\") pod \"openshift-lws-operator-fd8ccff4c-b58z8\" (UID: \"ec566262-22c4-4de3-826e-fcc0c7777ef0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"
Jul 31 14:22:44.516809 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.516784    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"
Jul 31 14:22:44.648565 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.648542    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8"]
Jul 31 14:22:44.651126 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:22:44.651096    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec566262_22c4_4de3_826e_fcc0c7777ef0.slice/crio-d97578b19b2ebfa483c5f018aaf0bb119a716fa8127bff465a412bbe1b8a1889 WatchSource:0}: Error finding container d97578b19b2ebfa483c5f018aaf0bb119a716fa8127bff465a412bbe1b8a1889: Status 404 returned error can't find the container with id d97578b19b2ebfa483c5f018aaf0bb119a716fa8127bff465a412bbe1b8a1889
Jul 31 14:22:44.786867 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:44.786778    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8" event={"ID":"ec566262-22c4-4de3-826e-fcc0c7777ef0","Type":"ContainerStarted","Data":"d97578b19b2ebfa483c5f018aaf0bb119a716fa8127bff465a412bbe1b8a1889"}
Jul 31 14:22:45.313542 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:45.313514    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:22:45.316200 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:45.316167    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:22:47.803572 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:47.803537    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mrt97" event={"ID":"1a09a135-e020-4174-87ff-ea7d49dbbb36","Type":"ContainerStarted","Data":"50d82c5ebf49f8d129e9ecbbc6147c2ff1eba1f6e63a02208ba977390d55696f"}
Jul 31 14:22:47.822503 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:47.822452    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mrt97" podStartSLOduration=1.658766418 podStartE2EDuration="5.822423866s" totalImagesPullingTime="4.163657448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:22:47.821015572 +0000 UTC m=+606.693578825" watchObservedRunningTime="2026-07-31 14:22:47.822423866 +0000 UTC m=+606.694987119"
Jul 31 14:22:59.854218 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:59.854183    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8" event={"ID":"ec566262-22c4-4de3-826e-fcc0c7777ef0","Type":"ContainerStarted","Data":"ef3c754b8cb02825e1363bd3feb031a2d16f1df7d2ac795a1fd0af3eaa662761"}
Jul 31 14:22:59.870973 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:22:59.870927    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-b58z8" podStartSLOduration=1.318331191 podStartE2EDuration="15.870913115s" totalImagesPullingTime="14.552581924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:22:59.869549571 +0000 UTC m=+618.742112825" watchObservedRunningTime="2026-07-31 14:22:59.870913115 +0000 UTC m=+618.743476368"
Jul 31 14:23:16.382738 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.382702    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"]
Jul 31 14:23:16.385205 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.385189    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.388576 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.388546    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 31 14:23:16.388576 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.388546    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 31 14:23:16.388785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.388630    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 31 14:23:16.388785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.388634    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2k6vh\""
Jul 31 14:23:16.388785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.388686    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 31 14:23:16.402568 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.402547    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"]
Jul 31 14:23:16.460237 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.460211    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.460381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.460256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.460381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.460319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.460454 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.460389    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.460454 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.460416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkmh\" (UniqueName: \"kubernetes.io/projected/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-kube-api-access-5lkmh\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.561266 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.561234    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.561433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.561270    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkmh\" (UniqueName: \"kubernetes.io/projected/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-kube-api-access-5lkmh\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.561433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.561319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.561433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.561348    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.561433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.561373    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.561652 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.561606    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-manifests\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.563571 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.563545    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-tmp\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.563815 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.563799    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.563881 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.563836    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-webhook-cert\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.569011 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.568981    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkmh\" (UniqueName: \"kubernetes.io/projected/67a9840e-4dcf-4ed4-8e17-9026c5b17cbe-kube-api-access-5lkmh\") pod \"opendatahub-operator-controller-manager-698c94c45b-k4dzm\" (UID: \"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe\") " pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.695615 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.695527    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:16.825991 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.825967    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"]
Jul 31 14:23:16.828177 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:16.828149    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a9840e_4dcf_4ed4_8e17_9026c5b17cbe.slice/crio-0e1246b398c9f2b07135f74fe4516994edf12de73aadfb9b454c511379f236aa WatchSource:0}: Error finding container 0e1246b398c9f2b07135f74fe4516994edf12de73aadfb9b454c511379f236aa: Status 404 returned error can't find the container with id 0e1246b398c9f2b07135f74fe4516994edf12de73aadfb9b454c511379f236aa
Jul 31 14:23:16.829925 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.829904    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:23:16.927404 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:16.927367    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm" event={"ID":"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe","Type":"ContainerStarted","Data":"0e1246b398c9f2b07135f74fe4516994edf12de73aadfb9b454c511379f236aa"}
Jul 31 14:23:20.949723 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:20.949681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm" event={"ID":"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe","Type":"ContainerStarted","Data":"3a896a7303670dfa7321a06a5d0c0d0e88d0cc9ba3fad2a8e45e171905a54b2a"}
Jul 31 14:23:21.955181 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:21.955147    2558 generic.go:356] "Generic (PLEG): container finished" podID="67a9840e-4dcf-4ed4-8e17-9026c5b17cbe" containerID="3a896a7303670dfa7321a06a5d0c0d0e88d0cc9ba3fad2a8e45e171905a54b2a" exitCode=0
Jul 31 14:23:21.955582 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:21.955200    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm" event={"ID":"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe","Type":"ContainerDied","Data":"3a896a7303670dfa7321a06a5d0c0d0e88d0cc9ba3fad2a8e45e171905a54b2a"}
Jul 31 14:23:22.961166 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:22.961125    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm" event={"ID":"67a9840e-4dcf-4ed4-8e17-9026c5b17cbe","Type":"ContainerStarted","Data":"fae4182b8d145c26a1d0871562ad2b2f10acff21e80f756deeffe085a86e4e6d"}
Jul 31 14:23:22.961549 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:22.961217    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:22.980604 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:22.980545    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm" podStartSLOduration=2.249175922 podStartE2EDuration="6.980525903s" totalImagesPullingTime="3.644700884s" totalInitContainerRuntime="1.086649097s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:22.978835534 +0000 UTC m=+641.851398807" watchObservedRunningTime="2026-07-31 14:23:22.980525903 +0000 UTC m=+641.853089220"
Jul 31 14:23:33.968544 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:33.968512    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-698c94c45b-k4dzm"
Jul 31 14:23:36.238894 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.238860    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g95vk"]
Jul 31 14:23:36.241663 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.241646    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:36.245458 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.245436    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 31 14:23:36.245870 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.245853    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-v9khg\""
Jul 31 14:23:36.256917 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.256881    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g95vk"]
Jul 31 14:23:36.335693 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.335650    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2td7\" (UniqueName: \"kubernetes.io/projected/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-kube-api-access-b2td7\") pod \"odh-model-controller-7998c67dc4-g95vk\" (UID: \"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:36.335869 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.335718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-cert\") pod \"odh-model-controller-7998c67dc4-g95vk\" (UID: \"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:36.436698 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.436659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-cert\") pod \"odh-model-controller-7998c67dc4-g95vk\" (UID: \"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:36.436865 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.436741    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2td7\" (UniqueName: \"kubernetes.io/projected/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-kube-api-access-b2td7\") pod \"odh-model-controller-7998c67dc4-g95vk\" (UID: \"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:36.436865 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:36.436801    2558 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 31 14:23:36.436936 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:36.436866    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-cert podName:b42a1bc2-e21f-44db-ab62-fb98ac94c8d1 nodeName:}" failed. No retries permitted until 2026-07-31 14:23:36.936850906 +0000 UTC m=+655.809414136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-cert") pod "odh-model-controller-7998c67dc4-g95vk" (UID: "b42a1bc2-e21f-44db-ab62-fb98ac94c8d1") : secret "odh-model-controller-webhook-cert" not found
Jul 31 14:23:36.446118 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.446089    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2td7\" (UniqueName: \"kubernetes.io/projected/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-kube-api-access-b2td7\") pod \"odh-model-controller-7998c67dc4-g95vk\" (UID: \"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:36.820553 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.820521    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ptcph"]
Jul 31 14:23:36.823342 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.823325    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:36.825725 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.825694    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 31 14:23:36.825836 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.825748    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-jvp96\""
Jul 31 14:23:36.833574 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.833534    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ptcph"]
Jul 31 14:23:36.840561 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.840533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c936181c-91ed-4e97-abc0-d63c2912a32a-tls-certs\") pod \"model-serving-api-54d89bd5fc-ptcph\" (UID: \"c936181c-91ed-4e97-abc0-d63c2912a32a\") " pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:36.840672 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.840574    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q64bp\" (UniqueName: \"kubernetes.io/projected/c936181c-91ed-4e97-abc0-d63c2912a32a-kube-api-access-q64bp\") pod \"model-serving-api-54d89bd5fc-ptcph\" (UID: \"c936181c-91ed-4e97-abc0-d63c2912a32a\") " pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:36.941310 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.941274    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c936181c-91ed-4e97-abc0-d63c2912a32a-tls-certs\") pod \"model-serving-api-54d89bd5fc-ptcph\" (UID: \"c936181c-91ed-4e97-abc0-d63c2912a32a\") " pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:36.941310 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.941304    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q64bp\" (UniqueName: \"kubernetes.io/projected/c936181c-91ed-4e97-abc0-d63c2912a32a-kube-api-access-q64bp\") pod \"model-serving-api-54d89bd5fc-ptcph\" (UID: \"c936181c-91ed-4e97-abc0-d63c2912a32a\") " pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:36.941548 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.941380    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-cert\") pod \"odh-model-controller-7998c67dc4-g95vk\" (UID: \"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:36.941548 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:36.941418    2558 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 31 14:23:36.941548 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:36.941497    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c936181c-91ed-4e97-abc0-d63c2912a32a-tls-certs podName:c936181c-91ed-4e97-abc0-d63c2912a32a nodeName:}" failed. No retries permitted until 2026-07-31 14:23:37.441478192 +0000 UTC m=+656.314041424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c936181c-91ed-4e97-abc0-d63c2912a32a-tls-certs") pod "model-serving-api-54d89bd5fc-ptcph" (UID: "c936181c-91ed-4e97-abc0-d63c2912a32a") : secret "model-serving-api-tls" not found
Jul 31 14:23:36.943751 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.943734    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b42a1bc2-e21f-44db-ab62-fb98ac94c8d1-cert\") pod \"odh-model-controller-7998c67dc4-g95vk\" (UID: \"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1\") " pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:36.949699 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:36.949675    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q64bp\" (UniqueName: \"kubernetes.io/projected/c936181c-91ed-4e97-abc0-d63c2912a32a-kube-api-access-q64bp\") pod \"model-serving-api-54d89bd5fc-ptcph\" (UID: \"c936181c-91ed-4e97-abc0-d63c2912a32a\") " pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:37.151376 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:37.151286    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:37.277719 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:37.277690    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-g95vk"]
Jul 31 14:23:37.279647 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:37.279617    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42a1bc2_e21f_44db_ab62_fb98ac94c8d1.slice/crio-16cdda0e24cbb2d78edab449cd9afe8a433481889d67f6f01e454a50b5ddfcdf WatchSource:0}: Error finding container 16cdda0e24cbb2d78edab449cd9afe8a433481889d67f6f01e454a50b5ddfcdf: Status 404 returned error can't find the container with id 16cdda0e24cbb2d78edab449cd9afe8a433481889d67f6f01e454a50b5ddfcdf
Jul 31 14:23:37.445701 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:37.445597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c936181c-91ed-4e97-abc0-d63c2912a32a-tls-certs\") pod \"model-serving-api-54d89bd5fc-ptcph\" (UID: \"c936181c-91ed-4e97-abc0-d63c2912a32a\") " pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:37.448068 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:37.448039    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c936181c-91ed-4e97-abc0-d63c2912a32a-tls-certs\") pod \"model-serving-api-54d89bd5fc-ptcph\" (UID: \"c936181c-91ed-4e97-abc0-d63c2912a32a\") " pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:37.735237 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:37.735150    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:37.860414 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:37.860387    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-ptcph"]
Jul 31 14:23:37.862276 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:37.862239    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc936181c_91ed_4e97_abc0_d63c2912a32a.slice/crio-4d978e16181dd2c167aceda618bb59046b068c964a6e85167e608ad3ddf94bd2 WatchSource:0}: Error finding container 4d978e16181dd2c167aceda618bb59046b068c964a6e85167e608ad3ddf94bd2: Status 404 returned error can't find the container with id 4d978e16181dd2c167aceda618bb59046b068c964a6e85167e608ad3ddf94bd2
Jul 31 14:23:38.028516 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.028427    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ptcph" event={"ID":"c936181c-91ed-4e97-abc0-d63c2912a32a","Type":"ContainerStarted","Data":"4d978e16181dd2c167aceda618bb59046b068c964a6e85167e608ad3ddf94bd2"}
Jul 31 14:23:38.029638 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.029611    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g95vk" event={"ID":"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1","Type":"ContainerStarted","Data":"16cdda0e24cbb2d78edab449cd9afe8a433481889d67f6f01e454a50b5ddfcdf"}
Jul 31 14:23:38.892306 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.892267    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"]
Jul 31 14:23:38.895987 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.895964    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:38.901264 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.901237    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 31 14:23:38.901971 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.901503    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 31 14:23:38.901971 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.901745    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 31 14:23:38.905152 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.904148    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 31 14:23:38.907053 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.906914    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"]
Jul 31 14:23:38.960175 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.959866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzzlj\" (UniqueName: \"kubernetes.io/projected/80f5291b-3486-44ad-8db4-d9ce9301f825-kube-api-access-pzzlj\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:38.960175 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.959975    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f5291b-3486-44ad-8db4-d9ce9301f825-tls-certs\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:38.960175 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:38.960056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80f5291b-3486-44ad-8db4-d9ce9301f825-tmp\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.060672 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.060640    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f5291b-3486-44ad-8db4-d9ce9301f825-tls-certs\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.060852 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.060692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80f5291b-3486-44ad-8db4-d9ce9301f825-tmp\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.060852 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.060718    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzzlj\" (UniqueName: \"kubernetes.io/projected/80f5291b-3486-44ad-8db4-d9ce9301f825-kube-api-access-pzzlj\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.060852 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:39.060806    2558 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 31 14:23:39.061042 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:39.060889    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80f5291b-3486-44ad-8db4-d9ce9301f825-tls-certs podName:80f5291b-3486-44ad-8db4-d9ce9301f825 nodeName:}" failed. No retries permitted until 2026-07-31 14:23:39.560865626 +0000 UTC m=+658.433428862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/80f5291b-3486-44ad-8db4-d9ce9301f825-tls-certs") pod "kube-auth-proxy-7c8b44b796-zwgwr" (UID: "80f5291b-3486-44ad-8db4-d9ce9301f825") : secret "kube-auth-proxy-tls" not found
Jul 31 14:23:39.062869 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.062851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80f5291b-3486-44ad-8db4-d9ce9301f825-tmp\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.071988 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.071939    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzzlj\" (UniqueName: \"kubernetes.io/projected/80f5291b-3486-44ad-8db4-d9ce9301f825-kube-api-access-pzzlj\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.566004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.565845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f5291b-3486-44ad-8db4-d9ce9301f825-tls-certs\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.569302 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.569280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80f5291b-3486-44ad-8db4-d9ce9301f825-tls-certs\") pod \"kube-auth-proxy-7c8b44b796-zwgwr\" (UID: \"80f5291b-3486-44ad-8db4-d9ce9301f825\") " pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.810757 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.810724    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"
Jul 31 14:23:39.857192 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.857117    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"]
Jul 31 14:23:39.860578 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.860557    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:39.863159 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.863106    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 31 14:23:39.863159 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.863107    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 31 14:23:39.863381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.863113    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 31 14:23:39.863443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.863414    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-l2qsb\""
Jul 31 14:23:39.863443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.863429    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 31 14:23:39.875336 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.875312    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"]
Jul 31 14:23:39.969888 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.969842    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d041270f-06a5-480c-acf6-9d24607f7a8d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:39.969888 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.969890    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:39.970378 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.970081    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pxt\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-kube-api-access-49pxt\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:39.970378 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.970123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:39.970378 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.970152    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:39.970378 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.970229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:39.970378 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:39.970260    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.071269 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.071228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.071456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.071277    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.071456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.071329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.071456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.071353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.071456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.071390    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d041270f-06a5-480c-acf6-9d24607f7a8d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.071456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.071423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.071732 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.071541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pxt\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-kube-api-access-49pxt\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.072108 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.072073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.074419 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.074377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d041270f-06a5-480c-acf6-9d24607f7a8d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.074532 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.074423    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.074532 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.074475    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.074647 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.074531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.081589 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.081565    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.082414 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.082369    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pxt\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-kube-api-access-49pxt\") pod \"istiod-openshift-gateway-7b7995d5cd-wplmd\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.171535 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.171441    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:40.858384 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.858353    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"]
Jul 31 14:23:40.863042 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:40.863004    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd041270f_06a5_480c_acf6_9d24607f7a8d.slice/crio-52b32a56daafd454aa3cd10fae68927d3bc765d6f22a11c7695fc098d1f20a3b WatchSource:0}: Error finding container 52b32a56daafd454aa3cd10fae68927d3bc765d6f22a11c7695fc098d1f20a3b: Status 404 returned error can't find the container with id 52b32a56daafd454aa3cd10fae68927d3bc765d6f22a11c7695fc098d1f20a3b
Jul 31 14:23:40.873916 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:40.873888    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr"]
Jul 31 14:23:40.875468 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:40.875447    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f5291b_3486_44ad_8db4_d9ce9301f825.slice/crio-3828c0896f0c5f7c2b7bce67b73f7a4d3a225b6486eeee00e49123db8c4a2a8b WatchSource:0}: Error finding container 3828c0896f0c5f7c2b7bce67b73f7a4d3a225b6486eeee00e49123db8c4a2a8b: Status 404 returned error can't find the container with id 3828c0896f0c5f7c2b7bce67b73f7a4d3a225b6486eeee00e49123db8c4a2a8b
Jul 31 14:23:41.046903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:41.046860    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-ptcph" event={"ID":"c936181c-91ed-4e97-abc0-d63c2912a32a","Type":"ContainerStarted","Data":"137adae09366e204d0ce39b9a9ffbae005ef1fad1de532504c3f98d9fb405b24"}
Jul 31 14:23:41.047408 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:41.047044    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:41.048689 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:41.048655    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-g95vk" event={"ID":"b42a1bc2-e21f-44db-ab62-fb98ac94c8d1","Type":"ContainerStarted","Data":"5d242f285d958ddd058877c5557daa47f111b530ea924cadcb2a96badcca496d"}
Jul 31 14:23:41.048829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:41.048744    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:41.049900 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:41.049873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr" event={"ID":"80f5291b-3486-44ad-8db4-d9ce9301f825","Type":"ContainerStarted","Data":"3828c0896f0c5f7c2b7bce67b73f7a4d3a225b6486eeee00e49123db8c4a2a8b"}
Jul 31 14:23:41.051143 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:41.051124    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd" event={"ID":"d041270f-06a5-480c-acf6-9d24607f7a8d","Type":"ContainerStarted","Data":"52b32a56daafd454aa3cd10fae68927d3bc765d6f22a11c7695fc098d1f20a3b"}
Jul 31 14:23:41.064141 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:41.064102    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-ptcph" podStartSLOduration=2.217617297 podStartE2EDuration="5.064088106s" totalImagesPullingTime="2.846470809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:41.062334998 +0000 UTC m=+659.934898253" watchObservedRunningTime="2026-07-31 14:23:41.064088106 +0000 UTC m=+659.936651338"
Jul 31 14:23:41.077768 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:41.077728    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-g95vk" podStartSLOduration=1.648772504 podStartE2EDuration="5.077716461s" totalImagesPullingTime="3.428943957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:41.076063225 +0000 UTC m=+659.948626480" watchObservedRunningTime="2026-07-31 14:23:41.077716461 +0000 UTC m=+659.950279771"
Jul 31 14:23:42.834232 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:42.834198    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 31 14:23:43.060438 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:43.060402    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr" event={"ID":"80f5291b-3486-44ad-8db4-d9ce9301f825","Type":"ContainerStarted","Data":"14fab89a39b9ab26b86fd2cb673132ccfe4a4455db5b5de9e9cdd656af0060e7"}
Jul 31 14:23:43.084421 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:43.084334    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c8b44b796-zwgwr" podStartSLOduration=3.130003933 podStartE2EDuration="5.084318282s" totalImagesPullingTime="1.954314349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:43.082452322 +0000 UTC m=+661.955015575" watchObservedRunningTime="2026-07-31 14:23:43.084318282 +0000 UTC m=+661.956881535"
Jul 31 14:23:45.094443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:45.094395    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:23:45.094729 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:45.094463    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:23:46.079806 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.079762    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd" event={"ID":"d041270f-06a5-480c-acf6-9d24607f7a8d","Type":"ContainerStarted","Data":"7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895"}
Jul 31 14:23:46.080002 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.079956    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:46.081798 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.081766    2558 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-wplmd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 31 14:23:46.081917 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.081828    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd" podUID="d041270f-06a5-480c-acf6-9d24607f7a8d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 31 14:23:46.118796 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.118742    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd" podStartSLOduration=2.889716123 podStartE2EDuration="7.118725221s" totalImagesPullingTime="4.229009098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:46.116630634 +0000 UTC m=+664.989193887" watchObservedRunningTime="2026-07-31 14:23:46.118725221 +0000 UTC m=+664.991288474"
Jul 31 14:23:46.318145 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.318108    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"]
Jul 31 14:23:46.322380 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.322353    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.325530 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.325501    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gv5h8\""
Jul 31 14:23:46.340075 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.340010    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"]
Jul 31 14:23:46.431295 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431257    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.431517 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.431517 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431372    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.431517 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431415    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.431517 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431475    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.431517 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431500    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.431777 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.431777 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431578    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.431777 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.431605    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwsb\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-kube-api-access-qhwsb\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.443259 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.443226    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"]
Jul 31 14:23:46.446397 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.446382    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:46.451817 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.451794    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 31 14:23:46.451967 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.451871    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7hht2\""
Jul 31 14:23:46.467132 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.465009    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"]
Jul 31 14:23:46.474981 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.474956    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"]
Jul 31 14:23:46.475142 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.475095    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:46.480108 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.479817    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 31 14:23:46.480108 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.479852    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kxpd5\""
Jul 31 14:23:46.482494 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.482466    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"]
Jul 31 14:23:46.532708 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.532676    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.532874 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.532718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4nh\" (UniqueName: \"kubernetes.io/projected/54829018-37a2-4699-af96-aea18ce0a6d3-kube-api-access-sg4nh\") pod \"kserve-controller-manager-d6f4b65b8-xzlhj\" (UID: \"54829018-37a2-4699-af96-aea18ce0a6d3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:46.532874 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.532849    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.532960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.532906    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.532960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.532948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533080 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.532978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9xz\" (UniqueName: \"kubernetes.io/projected/ee99d4c6-3d42-42cd-a5d2-e6a94b651804-kube-api-access-4t9xz\") pod \"llmisvc-controller-manager-777c6bd7d5-l65hb\" (UID: \"ee99d4c6-3d42-42cd-a5d2-e6a94b651804\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:46.533080 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533080 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533069    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwsb\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-kube-api-access-qhwsb\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533227 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533104    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee99d4c6-3d42-42cd-a5d2-e6a94b651804-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l65hb\" (UID: \"ee99d4c6-3d42-42cd-a5d2-e6a94b651804\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:46.533227 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533142    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54829018-37a2-4699-af96-aea18ce0a6d3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xzlhj\" (UID: \"54829018-37a2-4699-af96-aea18ce0a6d3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:46.533227 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533182    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533384 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533226    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533384 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533384 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533384 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533592 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533543    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.533761 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.533740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.535633 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.535613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.535814 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.535796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.545381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.545351    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.545493 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.545474    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwsb\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-kube-api-access-qhwsb\") pod \"data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.634109 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.634010    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9xz\" (UniqueName: \"kubernetes.io/projected/ee99d4c6-3d42-42cd-a5d2-e6a94b651804-kube-api-access-4t9xz\") pod \"llmisvc-controller-manager-777c6bd7d5-l65hb\" (UID: \"ee99d4c6-3d42-42cd-a5d2-e6a94b651804\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:46.634109 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.634079    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee99d4c6-3d42-42cd-a5d2-e6a94b651804-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l65hb\" (UID: \"ee99d4c6-3d42-42cd-a5d2-e6a94b651804\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:46.634327 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.634131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54829018-37a2-4699-af96-aea18ce0a6d3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xzlhj\" (UID: \"54829018-37a2-4699-af96-aea18ce0a6d3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:46.634327 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.634222    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4nh\" (UniqueName: \"kubernetes.io/projected/54829018-37a2-4699-af96-aea18ce0a6d3-kube-api-access-sg4nh\") pod \"kserve-controller-manager-d6f4b65b8-xzlhj\" (UID: \"54829018-37a2-4699-af96-aea18ce0a6d3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:46.634327 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:46.634270    2558 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 31 14:23:46.634478 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:46.634344    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54829018-37a2-4699-af96-aea18ce0a6d3-cert podName:54829018-37a2-4699-af96-aea18ce0a6d3 nodeName:}" failed. No retries permitted until 2026-07-31 14:23:47.134321609 +0000 UTC m=+666.006884840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/54829018-37a2-4699-af96-aea18ce0a6d3-cert") pod "kserve-controller-manager-d6f4b65b8-xzlhj" (UID: "54829018-37a2-4699-af96-aea18ce0a6d3") : secret "kserve-webhook-server-cert" not found
Jul 31 14:23:46.636382 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.636360    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:46.636586 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.636567    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee99d4c6-3d42-42cd-a5d2-e6a94b651804-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l65hb\" (UID: \"ee99d4c6-3d42-42cd-a5d2-e6a94b651804\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:46.649435 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.649410    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9xz\" (UniqueName: \"kubernetes.io/projected/ee99d4c6-3d42-42cd-a5d2-e6a94b651804-kube-api-access-4t9xz\") pod \"llmisvc-controller-manager-777c6bd7d5-l65hb\" (UID: \"ee99d4c6-3d42-42cd-a5d2-e6a94b651804\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:46.652162 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.652138    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4nh\" (UniqueName: \"kubernetes.io/projected/54829018-37a2-4699-af96-aea18ce0a6d3-kube-api-access-sg4nh\") pod \"kserve-controller-manager-d6f4b65b8-xzlhj\" (UID: \"54829018-37a2-4699-af96-aea18ce0a6d3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:46.741069 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.740687    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"]
Jul 31 14:23:46.745553 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.745523    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.769565 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.768564    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"]
Jul 31 14:23:46.787797 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.787758    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:46.820245 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.820210    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"]
Jul 31 14:23:46.823948 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:46.823920    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c2f732_6114_4791_86ef_04d1fd0cc0d5.slice/crio-5970d81626e2632d95479d9a322b87e07be23d69ac5651f7e68abcff34f647bc WatchSource:0}: Error finding container 5970d81626e2632d95479d9a322b87e07be23d69ac5651f7e68abcff34f647bc: Status 404 returned error can't find the container with id 5970d81626e2632d95479d9a322b87e07be23d69ac5651f7e68abcff34f647bc
Jul 31 14:23:46.836132 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836067    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa698e3-34cf-43d6-967b-492af65a098c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.836132 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7sw8\" (UniqueName: \"kubernetes.io/projected/4fa698e3-34cf-43d6-967b-492af65a098c-kube-api-access-c7sw8\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.836318 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836174    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa698e3-34cf-43d6-967b-492af65a098c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.836378 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.836428 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.836476 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836434    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.836530 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa698e3-34cf-43d6-967b-492af65a098c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.836603 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836587    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.836657 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.836616    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937421 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937326    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937421 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937402    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937645 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937433    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937645 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa698e3-34cf-43d6-967b-492af65a098c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937645 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937537    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937645 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937645 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937608    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa698e3-34cf-43d6-967b-492af65a098c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937645 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7sw8\" (UniqueName: \"kubernetes.io/projected/4fa698e3-34cf-43d6-967b-492af65a098c-kube-api-access-c7sw8\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.937974 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.937663    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa698e3-34cf-43d6-967b-492af65a098c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.938426 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.938134    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.938426 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.938386    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.939092 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.938450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.939092 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.938576    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.939092 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.939046    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa698e3-34cf-43d6-967b-492af65a098c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.940557 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.940531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa698e3-34cf-43d6-967b-492af65a098c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.941295 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.941274    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa698e3-34cf-43d6-967b-492af65a098c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.941666 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.941648    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"]
Jul 31 14:23:46.942491 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:46.942471    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podee99d4c6_3d42_42cd_a5d2_e6a94b651804.slice/crio-b69f54149e007d76ae9c644532936075e7b1116343f95be8ee6b8722407f9d3e WatchSource:0}: Error finding container b69f54149e007d76ae9c644532936075e7b1116343f95be8ee6b8722407f9d3e: Status 404 returned error can't find the container with id b69f54149e007d76ae9c644532936075e7b1116343f95be8ee6b8722407f9d3e
Jul 31 14:23:46.951849 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.951820    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa698e3-34cf-43d6-967b-492af65a098c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:46.952073 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:46.952055    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7sw8\" (UniqueName: \"kubernetes.io/projected/4fa698e3-34cf-43d6-967b-492af65a098c-kube-api-access-c7sw8\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c\" (UID: \"4fa698e3-34cf-43d6-967b-492af65a098c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:47.062436 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.062403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:47.086099 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.086067    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb" event={"ID":"ee99d4c6-3d42-42cd-a5d2-e6a94b651804","Type":"ContainerStarted","Data":"b69f54149e007d76ae9c644532936075e7b1116343f95be8ee6b8722407f9d3e"}
Jul 31 14:23:47.087411 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.087385    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" event={"ID":"f5c2f732-6114-4791-86ef-04d1fd0cc0d5","Type":"ContainerStarted","Data":"5970d81626e2632d95479d9a322b87e07be23d69ac5651f7e68abcff34f647bc"}
Jul 31 14:23:47.088235 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.088216    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:47.140139 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.140091    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54829018-37a2-4699-af96-aea18ce0a6d3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xzlhj\" (UID: \"54829018-37a2-4699-af96-aea18ce0a6d3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:47.143049 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.142994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54829018-37a2-4699-af96-aea18ce0a6d3-cert\") pod \"kserve-controller-manager-d6f4b65b8-xzlhj\" (UID: \"54829018-37a2-4699-af96-aea18ce0a6d3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:47.200715 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.200691    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"]
Jul 31 14:23:47.202768 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:47.202725    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa698e3_34cf_43d6_967b_492af65a098c.slice/crio-56e1f32531cd4c8fc98bd79f8ac004022b355a68bedad67baeb9d6d9680e89a9 WatchSource:0}: Error finding container 56e1f32531cd4c8fc98bd79f8ac004022b355a68bedad67baeb9d6d9680e89a9: Status 404 returned error can't find the container with id 56e1f32531cd4c8fc98bd79f8ac004022b355a68bedad67baeb9d6d9680e89a9
Jul 31 14:23:47.357046 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.356996    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:47.515204 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:47.515178    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"]
Jul 31 14:23:47.517184 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:23:47.517155    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54829018_37a2_4699_af96_aea18ce0a6d3.slice/crio-4ae49a699757e91e8c964d39654b3d596b3bfbc4677f75fdd40283e88f3e988a WatchSource:0}: Error finding container 4ae49a699757e91e8c964d39654b3d596b3bfbc4677f75fdd40283e88f3e988a: Status 404 returned error can't find the container with id 4ae49a699757e91e8c964d39654b3d596b3bfbc4677f75fdd40283e88f3e988a
Jul 31 14:23:48.093151 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:48.093117    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c" event={"ID":"4fa698e3-34cf-43d6-967b-492af65a098c","Type":"ContainerStarted","Data":"56e1f32531cd4c8fc98bd79f8ac004022b355a68bedad67baeb9d6d9680e89a9"}
Jul 31 14:23:48.094180 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:48.094154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj" event={"ID":"54829018-37a2-4699-af96-aea18ce0a6d3","Type":"ContainerStarted","Data":"4ae49a699757e91e8c964d39654b3d596b3bfbc4677f75fdd40283e88f3e988a"}
Jul 31 14:23:49.997746 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:49.997711    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"]
Jul 31 14:23:49.998188 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:49.997929    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd" podUID="d041270f-06a5-480c-acf6-9d24607f7a8d" containerName="discovery" containerID="cri-o://7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895" gracePeriod=30
Jul 31 14:23:50.088602 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.088567    2558 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-wplmd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:8080/ready\": dial tcp 10.134.0.32:8080: connect: connection refused" start-of-body=
Jul 31 14:23:50.088783 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.088626    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd" podUID="d041270f-06a5-480c-acf6-9d24607f7a8d" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.32:8080/ready\": dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 31 14:23:50.900193 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.900168    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:50.978138 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.978103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-token\") pod \"d041270f-06a5-480c-acf6-9d24607f7a8d\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") "
Jul 31 14:23:50.978318 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.978166    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-ca-configmap\") pod \"d041270f-06a5-480c-acf6-9d24607f7a8d\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") "
Jul 31 14:23:50.978318 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.978286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-dns-cert\") pod \"d041270f-06a5-480c-acf6-9d24607f7a8d\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") "
Jul 31 14:23:50.978785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.978755    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-kubeconfig\") pod \"d041270f-06a5-480c-acf6-9d24607f7a8d\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") "
Jul 31 14:23:50.978904 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.978818    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d041270f-06a5-480c-acf6-9d24607f7a8d-local-certs\") pod \"d041270f-06a5-480c-acf6-9d24607f7a8d\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") "
Jul 31 14:23:50.978904 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.978848    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d041270f-06a5-480c-acf6-9d24607f7a8d" (UID: "d041270f-06a5-480c-acf6-9d24607f7a8d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:23:50.978904 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.978863    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-cacerts\") pod \"d041270f-06a5-480c-acf6-9d24607f7a8d\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") "
Jul 31 14:23:50.979085 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.978915    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49pxt\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-kube-api-access-49pxt\") pod \"d041270f-06a5-480c-acf6-9d24607f7a8d\" (UID: \"d041270f-06a5-480c-acf6-9d24607f7a8d\") "
Jul 31 14:23:50.981973 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.981910    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-kube-api-access-49pxt" (OuterVolumeSpecName: "kube-api-access-49pxt") pod "d041270f-06a5-480c-acf6-9d24607f7a8d" (UID: "d041270f-06a5-480c-acf6-9d24607f7a8d"). InnerVolumeSpecName "kube-api-access-49pxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:23:50.982121 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.982053    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d041270f-06a5-480c-acf6-9d24607f7a8d" (UID: "d041270f-06a5-480c-acf6-9d24607f7a8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:23:50.982185 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.982154    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d041270f-06a5-480c-acf6-9d24607f7a8d" (UID: "d041270f-06a5-480c-acf6-9d24607f7a8d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:23:50.982185 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.982161    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d041270f-06a5-480c-acf6-9d24607f7a8d-local-certs" (OuterVolumeSpecName: "local-certs") pod "d041270f-06a5-480c-acf6-9d24607f7a8d" (UID: "d041270f-06a5-480c-acf6-9d24607f7a8d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:23:50.985044 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.982423    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:23:50.985044 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.982456    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-ca-configmap\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:23:50.985044 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.982473    2558 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-kubeconfig\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:23:50.985044 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.982488    2558 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d041270f-06a5-480c-acf6-9d24607f7a8d-local-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:23:50.985044 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.982502    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49pxt\" (UniqueName: \"kubernetes.io/projected/d041270f-06a5-480c-acf6-9d24607f7a8d-kube-api-access-49pxt\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:23:50.985044 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.984191    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d041270f-06a5-480c-acf6-9d24607f7a8d" (UID: "d041270f-06a5-480c-acf6-9d24607f7a8d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:23:50.989865 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:50.989831    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-cacerts" (OuterVolumeSpecName: "cacerts") pod "d041270f-06a5-480c-acf6-9d24607f7a8d" (UID: "d041270f-06a5-480c-acf6-9d24607f7a8d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:23:51.083042 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.082994    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-istio-csr-dns-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:23:51.083480 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.083061    2558 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d041270f-06a5-480c-acf6-9d24607f7a8d-cacerts\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:23:51.121363 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.121322    2558 generic.go:356] "Generic (PLEG): container finished" podID="d041270f-06a5-480c-acf6-9d24607f7a8d" containerID="7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895" exitCode=0
Jul 31 14:23:51.121536 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.121411    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"
Jul 31 14:23:51.121536 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.121411    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd" event={"ID":"d041270f-06a5-480c-acf6-9d24607f7a8d","Type":"ContainerDied","Data":"7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895"}
Jul 31 14:23:51.121536 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.121521    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd" event={"ID":"d041270f-06a5-480c-acf6-9d24607f7a8d","Type":"ContainerDied","Data":"52b32a56daafd454aa3cd10fae68927d3bc765d6f22a11c7695fc098d1f20a3b"}
Jul 31 14:23:51.121700 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.121554    2558 scope.go:117] "RemoveContainer" containerID="7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895"
Jul 31 14:23:51.159016 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.158984    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"]
Jul 31 14:23:51.162406 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.162378    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wplmd"]
Jul 31 14:23:51.580804 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:51.580761    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d041270f-06a5-480c-acf6-9d24607f7a8d" path="/var/lib/kubelet/pods/d041270f-06a5-480c-acf6-9d24607f7a8d/volumes"
Jul 31 14:23:52.057512 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.057485    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-g95vk"
Jul 31 14:23:52.059392 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.059374    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-ptcph"
Jul 31 14:23:52.088129 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.088100    2558 scope.go:117] "RemoveContainer" containerID="7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895"
Jul 31 14:23:52.088603 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:23:52.088537    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895\": container with ID starting with 7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895 not found: ID does not exist" containerID="7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895"
Jul 31 14:23:52.088603 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.088575    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895"} err="failed to get container status \"7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895\": rpc error: code = NotFound desc = could not find container \"7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895\": container with ID starting with 7bc8a33ec92a77d40528eee691201ee1de19ac438727e1bc5d4de2577e14c895 not found: ID does not exist"
Jul 31 14:23:52.611834 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.611793    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:23:52.612004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.611884    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:23:52.612004 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.611924    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:23:52.617317 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.617292    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:23:52.617407 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.617363    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:23:52.617407 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:52.617394    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:23:53.134401 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.134366    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c" event={"ID":"4fa698e3-34cf-43d6-967b-492af65a098c","Type":"ContainerStarted","Data":"44aba154833e8bbc389537fe8f5c64e5f3a40b0567e82547dfc9bcd973036d36"}
Jul 31 14:23:53.135922 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.135891    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" event={"ID":"f5c2f732-6114-4791-86ef-04d1fd0cc0d5","Type":"ContainerStarted","Data":"be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622"}
Jul 31 14:23:53.137433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.137407    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj" event={"ID":"54829018-37a2-4699-af96-aea18ce0a6d3","Type":"ContainerStarted","Data":"ca3e6117878747d3c4d29ed53c9823d995fe6f23af3be95f94bd96c2dda89f9e"}
Jul 31 14:23:53.137548 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.137498    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:23:53.138822 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.138801    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb" event={"ID":"ee99d4c6-3d42-42cd-a5d2-e6a94b651804","Type":"ContainerStarted","Data":"bb1cec2128e050cd24e381d6f2a3f2f2f6c6e5b614de2665324da37a86eb21f9"}
Jul 31 14:23:53.138902 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.138841    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:23:53.158511 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.158464    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c" podStartSLOduration=1.751739743 podStartE2EDuration="7.158447947s" totalImagesPullingTime="5.406708204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:53.158251824 +0000 UTC m=+672.030815079" watchObservedRunningTime="2026-07-31 14:23:53.158447947 +0000 UTC m=+672.031011177"
Jul 31 14:23:53.182386 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.182338    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" podStartSLOduration=1.391474424 podStartE2EDuration="7.182324487s" totalImagesPullingTime="5.790850063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:53.181270631 +0000 UTC m=+672.053833898" watchObservedRunningTime="2026-07-31 14:23:53.182324487 +0000 UTC m=+672.054887739"
Jul 31 14:23:53.199837 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.199790    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb" podStartSLOduration=2.027920523 podStartE2EDuration="7.199777988s" totalImagesPullingTime="5.171857465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:53.198408532 +0000 UTC m=+672.070971785" watchObservedRunningTime="2026-07-31 14:23:53.199777988 +0000 UTC m=+672.072341241"
Jul 31 14:23:53.217542 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.217503    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj" podStartSLOduration=2.625958599 podStartE2EDuration="7.217487214s" totalImagesPullingTime="4.591528615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:23:53.21576228 +0000 UTC m=+672.088325530" watchObservedRunningTime="2026-07-31 14:23:53.217487214 +0000 UTC m=+672.090050467"
Jul 31 14:23:53.636772 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.636734    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:23:53.638111 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.638090    2558 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 31 14:23:53.638176 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:53.638137    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" podUID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 31 14:23:54.062702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:54.062666    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:54.067874 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:54.067850    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:54.145139 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:54.145112    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:54.146143 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:54.146117    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c"
Jul 31 14:23:54.255577 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:54.255542    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"]
Jul 31 14:23:54.637818 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:54.637776    2558 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 31 14:23:54.637997 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:54.637839    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" podUID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 31 14:23:55.636733 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:55.636695    2558 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 31 14:23:55.637321 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:55.636754    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" podUID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 31 14:23:56.154205 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:23:56.154155    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" podUID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" containerName="istio-proxy" containerID="cri-o://be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622" gracePeriod=30
Jul 31 14:24:01.402755 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.402730    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:24:01.479266 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479188    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-envoy\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479266 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479259    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-certs\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479286    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istiod-ca-cert\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479316    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-data\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479358    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-credential-socket\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479383    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-socket\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479413    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-token\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479436    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhwsb\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-kube-api-access-qhwsb\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479472    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-podinfo\") pod \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\" (UID: \"f5c2f732-6114-4791-86ef-04d1fd0cc0d5\") "
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479537    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479661    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479700    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-data" (OuterVolumeSpecName: "istio-data") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479721    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479731    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479862    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479882    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479896    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:01.479903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479908    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:01.480252 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.479921    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:01.481516 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.481484    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:24:01.481641 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.481561    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-token" (OuterVolumeSpecName: "istio-token") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:24:01.481988 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.481969    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 14:24:01.482081 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.481995    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-kube-api-access-qhwsb" (OuterVolumeSpecName: "kube-api-access-qhwsb") pod "f5c2f732-6114-4791-86ef-04d1fd0cc0d5" (UID: "f5c2f732-6114-4791-86ef-04d1fd0cc0d5"). InnerVolumeSpecName "kube-api-access-qhwsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:24:01.580642 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.580610    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:01.580642 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.580636    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhwsb\" (UniqueName: \"kubernetes.io/projected/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-kube-api-access-qhwsb\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:01.580642 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.580646    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:01.580869 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:01.580655    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5c2f732-6114-4791-86ef-04d1fd0cc0d5-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:24:02.184747 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.184712    2558 generic.go:356] "Generic (PLEG): container finished" podID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" containerID="be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622" exitCode=0
Jul 31 14:24:02.184915 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.184754    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"
Jul 31 14:24:02.184915 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.184780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" event={"ID":"f5c2f732-6114-4791-86ef-04d1fd0cc0d5","Type":"ContainerDied","Data":"be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622"}
Jul 31 14:24:02.184915 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.184822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4" event={"ID":"f5c2f732-6114-4791-86ef-04d1fd0cc0d5","Type":"ContainerDied","Data":"5970d81626e2632d95479d9a322b87e07be23d69ac5651f7e68abcff34f647bc"}
Jul 31 14:24:02.184915 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.184844    2558 scope.go:117] "RemoveContainer" containerID="be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622"
Jul 31 14:24:02.193760 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.193741    2558 scope.go:117] "RemoveContainer" containerID="be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622"
Jul 31 14:24:02.194012 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:24:02.193994    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622\": container with ID starting with be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622 not found: ID does not exist" containerID="be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622"
Jul 31 14:24:02.194164 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.194061    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622"} err="failed to get container status \"be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622\": rpc error: code = NotFound desc = could not find container \"be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622\": container with ID starting with be86225b440d507a034eb3d22b226d3d73591468ef66ee1a982e3dd81dc00622 not found: ID does not exist"
Jul 31 14:24:02.202615 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.202595    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"]
Jul 31 14:24:02.209243 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:02.209224    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-fddd6c88c-x8vh4"]
Jul 31 14:24:03.579433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:03.579401    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" path="/var/lib/kubelet/pods/f5c2f732-6114-4791-86ef-04d1fd0cc0d5/volumes"
Jul 31 14:24:24.147338 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:24.147307    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l65hb"
Jul 31 14:24:24.150096 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:24.150072    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xzlhj"
Jul 31 14:24:51.603264 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.603230    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-cl6tm"]
Jul 31 14:24:51.603653 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.603578    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" containerName="istio-proxy"
Jul 31 14:24:51.603653 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.603591    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" containerName="istio-proxy"
Jul 31 14:24:51.603653 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.603624    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d041270f-06a5-480c-acf6-9d24607f7a8d" containerName="discovery"
Jul 31 14:24:51.603653 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.603633    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d041270f-06a5-480c-acf6-9d24607f7a8d" containerName="discovery"
Jul 31 14:24:51.603789 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.603712    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d041270f-06a5-480c-acf6-9d24607f7a8d" containerName="discovery"
Jul 31 14:24:51.603789 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.603723    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c2f732-6114-4791-86ef-04d1fd0cc0d5" containerName="istio-proxy"
Jul 31 14:24:51.606413 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.606397    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm"
Jul 31 14:24:51.608932 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.608907    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 31 14:24:51.609078 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.608912    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 31 14:24:51.609224 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.609211    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-vzt5d\""
Jul 31 14:24:51.618738 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.618716    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-cl6tm"]
Jul 31 14:24:51.703998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.703963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnxhn\" (UniqueName: \"kubernetes.io/projected/0f52ba2d-1e9b-44c3-816e-b257cdbca654-kube-api-access-tnxhn\") pod \"authorino-operator-68d88948d8-cl6tm\" (UID: \"0f52ba2d-1e9b-44c3-816e-b257cdbca654\") " pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm"
Jul 31 14:24:51.805329 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.805291    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxhn\" (UniqueName: \"kubernetes.io/projected/0f52ba2d-1e9b-44c3-816e-b257cdbca654-kube-api-access-tnxhn\") pod \"authorino-operator-68d88948d8-cl6tm\" (UID: \"0f52ba2d-1e9b-44c3-816e-b257cdbca654\") " pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm"
Jul 31 14:24:51.814548 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.814523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxhn\" (UniqueName: \"kubernetes.io/projected/0f52ba2d-1e9b-44c3-816e-b257cdbca654-kube-api-access-tnxhn\") pod \"authorino-operator-68d88948d8-cl6tm\" (UID: \"0f52ba2d-1e9b-44c3-816e-b257cdbca654\") " pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm"
Jul 31 14:24:51.917393 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:51.917306    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm"
Jul 31 14:24:52.050690 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:52.050656    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-cl6tm"]
Jul 31 14:24:52.053005 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:24:52.052976    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f52ba2d_1e9b_44c3_816e_b257cdbca654.slice/crio-a1779ffaff0ad0f5d58b6dfe11e7eef82bd50b63ec24719e325dc153be1f583f WatchSource:0}: Error finding container a1779ffaff0ad0f5d58b6dfe11e7eef82bd50b63ec24719e325dc153be1f583f: Status 404 returned error can't find the container with id a1779ffaff0ad0f5d58b6dfe11e7eef82bd50b63ec24719e325dc153be1f583f
Jul 31 14:24:52.425986 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:52.425947    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm" event={"ID":"0f52ba2d-1e9b-44c3-816e-b257cdbca654","Type":"ContainerStarted","Data":"a1779ffaff0ad0f5d58b6dfe11e7eef82bd50b63ec24719e325dc153be1f583f"}
Jul 31 14:24:53.942667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:53.942623    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"]
Jul 31 14:24:53.945569 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:53.945545    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"
Jul 31 14:24:53.948829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:53.948807    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 31 14:24:53.948990 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:53.948967    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-n6z6l\""
Jul 31 14:24:53.955677 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:53.954602    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"]
Jul 31 14:24:54.027546 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:54.027514    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2rsw\" (UniqueName: \"kubernetes.io/projected/a1852a52-681d-48a7-9e03-564888a92c53-kube-api-access-l2rsw\") pod \"dns-operator-controller-manager-5985564776-vvg9h\" (UID: \"a1852a52-681d-48a7-9e03-564888a92c53\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"
Jul 31 14:24:54.128775 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:54.128731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2rsw\" (UniqueName: \"kubernetes.io/projected/a1852a52-681d-48a7-9e03-564888a92c53-kube-api-access-l2rsw\") pod \"dns-operator-controller-manager-5985564776-vvg9h\" (UID: \"a1852a52-681d-48a7-9e03-564888a92c53\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"
Jul 31 14:24:54.142219 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:54.142190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2rsw\" (UniqueName: \"kubernetes.io/projected/a1852a52-681d-48a7-9e03-564888a92c53-kube-api-access-l2rsw\") pod \"dns-operator-controller-manager-5985564776-vvg9h\" (UID: \"a1852a52-681d-48a7-9e03-564888a92c53\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"
Jul 31 14:24:54.262314 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:54.262227    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"
Jul 31 14:24:54.411275 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:54.411250    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"]
Jul 31 14:24:55.296828 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:24:55.296790    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1852a52_681d_48a7_9e03_564888a92c53.slice/crio-3c22a9016e4051b90a69c700d31a3950d4ee81d60475b5682eabaa0b2c307162 WatchSource:0}: Error finding container 3c22a9016e4051b90a69c700d31a3950d4ee81d60475b5682eabaa0b2c307162: Status 404 returned error can't find the container with id 3c22a9016e4051b90a69c700d31a3950d4ee81d60475b5682eabaa0b2c307162
Jul 31 14:24:55.442350 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:55.442313    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h" event={"ID":"a1852a52-681d-48a7-9e03-564888a92c53","Type":"ContainerStarted","Data":"3c22a9016e4051b90a69c700d31a3950d4ee81d60475b5682eabaa0b2c307162"}
Jul 31 14:24:55.443902 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:55.443871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm" event={"ID":"0f52ba2d-1e9b-44c3-816e-b257cdbca654","Type":"ContainerStarted","Data":"f2b7e5bcd85f476082dd97817a5cfb7789490e64a39251ec70ea0287b51ddf59"}
Jul 31 14:24:55.444049 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:55.443964    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm"
Jul 31 14:24:55.470301 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:55.470252    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm" podStartSLOduration=1.193905333 podStartE2EDuration="4.470235622s" totalImagesPullingTime="3.276330289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:24:55.46895177 +0000 UTC m=+734.341515025" watchObservedRunningTime="2026-07-31 14:24:55.470235622 +0000 UTC m=+734.342798875"
Jul 31 14:24:56.252544 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.252506    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"]
Jul 31 14:24:56.255553 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.255527    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:24:56.259156 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.259130    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tsfjv\""
Jul 31 14:24:56.271323 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.271297    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"]
Jul 31 14:24:56.349088 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.349052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7xmc\" (UniqueName: \"kubernetes.io/projected/efe7928e-4412-4c73-b10a-d6052ddbb8ba-kube-api-access-x7xmc\") pod \"limitador-operator-controller-manager-6c549b6d85-7jtlp\" (UID: \"efe7928e-4412-4c73-b10a-d6052ddbb8ba\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:24:56.449619 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.449584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7xmc\" (UniqueName: \"kubernetes.io/projected/efe7928e-4412-4c73-b10a-d6052ddbb8ba-kube-api-access-x7xmc\") pod \"limitador-operator-controller-manager-6c549b6d85-7jtlp\" (UID: \"efe7928e-4412-4c73-b10a-d6052ddbb8ba\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:24:56.458396 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.458366    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7xmc\" (UniqueName: \"kubernetes.io/projected/efe7928e-4412-4c73-b10a-d6052ddbb8ba-kube-api-access-x7xmc\") pod \"limitador-operator-controller-manager-6c549b6d85-7jtlp\" (UID: \"efe7928e-4412-4c73-b10a-d6052ddbb8ba\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:24:56.568276 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.568242    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:24:56.720534 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:56.720507    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"]
Jul 31 14:24:56.992582 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:24:56.992502    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe7928e_4412_4c73_b10a_d6052ddbb8ba.slice/crio-30fb1b03cb5c0d7de9c942112fca1f48c6c0df53b8c199992a4be82b8ed32b8c WatchSource:0}: Error finding container 30fb1b03cb5c0d7de9c942112fca1f48c6c0df53b8c199992a4be82b8ed32b8c: Status 404 returned error can't find the container with id 30fb1b03cb5c0d7de9c942112fca1f48c6c0df53b8c199992a4be82b8ed32b8c
Jul 31 14:24:57.457196 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:57.457158    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h" event={"ID":"a1852a52-681d-48a7-9e03-564888a92c53","Type":"ContainerStarted","Data":"bd1155539b00ce1aa1d56c50309c49f544760b6adffaeab28879c73908037052"}
Jul 31 14:24:57.457646 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:57.457280    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"
Jul 31 14:24:57.458434 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:57.458414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp" event={"ID":"efe7928e-4412-4c73-b10a-d6052ddbb8ba","Type":"ContainerStarted","Data":"30fb1b03cb5c0d7de9c942112fca1f48c6c0df53b8c199992a4be82b8ed32b8c"}
Jul 31 14:24:57.478014 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:57.477971    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h" podStartSLOduration=2.746654629 podStartE2EDuration="4.477956178s" totalImagesPullingTime="1.731301549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:24:57.477540755 +0000 UTC m=+736.350104010" watchObservedRunningTime="2026-07-31 14:24:57.477956178 +0000 UTC m=+736.350519433"
Jul 31 14:24:59.471558 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:59.471514    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp" event={"ID":"efe7928e-4412-4c73-b10a-d6052ddbb8ba","Type":"ContainerStarted","Data":"10ba55c29a6396c00f28a0a00d43fc26a3182718f0c454355780ae9f1a0495c1"}
Jul 31 14:24:59.471942 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:59.471597    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:24:59.487965 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:24:59.487917    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp" podStartSLOduration=1.526628837 podStartE2EDuration="3.487899705s" totalImagesPullingTime="1.961270868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:24:59.487525709 +0000 UTC m=+738.360088964" watchObservedRunningTime="2026-07-31 14:24:59.487899705 +0000 UTC m=+738.360462958"
Jul 31 14:25:00.316552 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.316512    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"]
Jul 31 14:25:00.320003 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.319979    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:00.324703 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.324685    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q27fh\""
Jul 31 14:25:00.337996 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.337735    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"]
Jul 31 14:25:00.488189 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.488154    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhsw9\" (UniqueName: \"kubernetes.io/projected/a0704714-6474-4ffe-bced-21bacf5ccce6-kube-api-access-jhsw9\") pod \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" (UID: \"a0704714-6474-4ffe-bced-21bacf5ccce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:00.488564 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.488219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0704714-6474-4ffe-bced-21bacf5ccce6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" (UID: \"a0704714-6474-4ffe-bced-21bacf5ccce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:00.588900 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.588818    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhsw9\" (UniqueName: \"kubernetes.io/projected/a0704714-6474-4ffe-bced-21bacf5ccce6-kube-api-access-jhsw9\") pod \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" (UID: \"a0704714-6474-4ffe-bced-21bacf5ccce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:00.588900 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.588884    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0704714-6474-4ffe-bced-21bacf5ccce6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" (UID: \"a0704714-6474-4ffe-bced-21bacf5ccce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:00.589235 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.589218    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0704714-6474-4ffe-bced-21bacf5ccce6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" (UID: \"a0704714-6474-4ffe-bced-21bacf5ccce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:00.601928 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.601903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhsw9\" (UniqueName: \"kubernetes.io/projected/a0704714-6474-4ffe-bced-21bacf5ccce6-kube-api-access-jhsw9\") pod \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" (UID: \"a0704714-6474-4ffe-bced-21bacf5ccce6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:00.629850 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.629824    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:00.764951 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:00.764925    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"]
Jul 31 14:25:00.766738 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:25:00.766695    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0704714_6474_4ffe_bced_21bacf5ccce6.slice/crio-11d87a1c1b3598b532a968deb2c0b7b4cdc2046ac2cddee30116a28351405d75 WatchSource:0}: Error finding container 11d87a1c1b3598b532a968deb2c0b7b4cdc2046ac2cddee30116a28351405d75: Status 404 returned error can't find the container with id 11d87a1c1b3598b532a968deb2c0b7b4cdc2046ac2cddee30116a28351405d75
Jul 31 14:25:01.482799 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:01.482767    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7" event={"ID":"a0704714-6474-4ffe-bced-21bacf5ccce6","Type":"ContainerStarted","Data":"11d87a1c1b3598b532a968deb2c0b7b4cdc2046ac2cddee30116a28351405d75"}
Jul 31 14:25:06.452922 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:06.452885    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-cl6tm"
Jul 31 14:25:06.513568 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:06.513535    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7" event={"ID":"a0704714-6474-4ffe-bced-21bacf5ccce6","Type":"ContainerStarted","Data":"63afdae2fc7516a16cca0e8ebb4f77cec8bf44f6beda4531d9e58972a00636aa"}
Jul 31 14:25:06.513760 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:06.513665    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:06.534746 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:06.534702    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7" podStartSLOduration=1.821480508 podStartE2EDuration="6.534684734s" totalImagesPullingTime="4.713204226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:25:06.532896076 +0000 UTC m=+745.405459330" watchObservedRunningTime="2026-07-31 14:25:06.534684734 +0000 UTC m=+745.407247989"
Jul 31 14:25:08.467480 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:08.467453    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-vvg9h"
Jul 31 14:25:10.479479 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:10.479452    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:25:17.521040 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:17.520993    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:18.537289 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.537258    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"]
Jul 31 14:25:18.540960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.540944    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:18.553176 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.553151    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"]
Jul 31 14:25:18.553466 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.553445    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f000a-48c8-4067-89f9-7e22eab913d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" (UID: \"795f000a-48c8-4067-89f9-7e22eab913d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:18.553521 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.553495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lntcw\" (UniqueName: \"kubernetes.io/projected/795f000a-48c8-4067-89f9-7e22eab913d3-kube-api-access-lntcw\") pod \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" (UID: \"795f000a-48c8-4067-89f9-7e22eab913d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:18.654825 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.654791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f000a-48c8-4067-89f9-7e22eab913d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" (UID: \"795f000a-48c8-4067-89f9-7e22eab913d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:18.655012 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.654862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lntcw\" (UniqueName: \"kubernetes.io/projected/795f000a-48c8-4067-89f9-7e22eab913d3-kube-api-access-lntcw\") pod \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" (UID: \"795f000a-48c8-4067-89f9-7e22eab913d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:18.655229 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.655209    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f000a-48c8-4067-89f9-7e22eab913d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" (UID: \"795f000a-48c8-4067-89f9-7e22eab913d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:18.665562 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.665539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lntcw\" (UniqueName: \"kubernetes.io/projected/795f000a-48c8-4067-89f9-7e22eab913d3-kube-api-access-lntcw\") pod \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" (UID: \"795f000a-48c8-4067-89f9-7e22eab913d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:18.851532 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.851500    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:18.988579 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:18.988555    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"]
Jul 31 14:25:18.990363 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:25:18.990331    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795f000a_48c8_4067_89f9_7e22eab913d3.slice/crio-6d77ea4aeb3849bea37bcdce7723f95a62af53eadebfe43e6a025fcbffbc34c9 WatchSource:0}: Error finding container 6d77ea4aeb3849bea37bcdce7723f95a62af53eadebfe43e6a025fcbffbc34c9: Status 404 returned error can't find the container with id 6d77ea4aeb3849bea37bcdce7723f95a62af53eadebfe43e6a025fcbffbc34c9
Jul 31 14:25:19.336205 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.335304    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"]
Jul 31 14:25:19.336205 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.335530    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" containerName="manager" containerID="cri-o://63afdae2fc7516a16cca0e8ebb4f77cec8bf44f6beda4531d9e58972a00636aa" gracePeriod=2
Jul 31 14:25:19.338288 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:19.338248    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:19.338428 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.338412    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"]
Jul 31 14:25:19.350822 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.350792    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"]
Jul 31 14:25:19.360204 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.360167    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"]
Jul 31 14:25:19.361554 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.361530    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" containerName="manager"
Jul 31 14:25:19.361704 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.361684    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" containerName="manager"
Jul 31 14:25:19.361826 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.361800    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" containerName="manager"
Jul 31 14:25:19.366382 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.366356    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"]
Jul 31 14:25:19.366500 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.366464    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:19.373741 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.373272    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"]
Jul 31 14:25:19.373861 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.373842    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" containerName="manager"
Jul 31 14:25:19.373861 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.373860    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" containerName="manager"
Jul 31 14:25:19.373979 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.373971    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" containerName="manager"
Jul 31 14:25:19.377440 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.377420    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"]
Jul 31 14:25:19.377593 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.377543    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:19.377664 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.377614    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp" podUID="efe7928e-4412-4c73-b10a-d6052ddbb8ba" containerName="manager" containerID="cri-o://10ba55c29a6396c00f28a0a00d43fc26a3182718f0c454355780ae9f1a0495c1" gracePeriod=2
Jul 31 14:25:19.379431 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.379410    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"]
Jul 31 14:25:19.386673 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.386652    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"]
Jul 31 14:25:19.393005 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.392986    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"]
Jul 31 14:25:19.394948 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:19.394925    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:19.397237 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:19.397216    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-7jtlp\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="efe7928e-4412-4c73-b10a-d6052ddbb8ba" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:25:19.402145 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.402126    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"]
Jul 31 14:25:19.402520 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.402507    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe7928e-4412-4c73-b10a-d6052ddbb8ba" containerName="manager"
Jul 31 14:25:19.402565 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.402522    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe7928e-4412-4c73-b10a-d6052ddbb8ba" containerName="manager"
Jul 31 14:25:19.402599 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.402593    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe7928e-4412-4c73-b10a-d6052ddbb8ba" containerName="manager"
Jul 31 14:25:19.405493 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.405478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"
Jul 31 14:25:19.416046 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.416000    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"]
Jul 31 14:25:19.424453 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:19.424421    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:19.449998 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:19.449964    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-vbhp7\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:19.452077 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:19.452050    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-7jtlp\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="efe7928e-4412-4c73-b10a-d6052ddbb8ba" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:25:19.461484 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.461453    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wtjn\" (UniqueName: \"kubernetes.io/projected/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-kube-api-access-6wtjn\") pod \"kuadrant-operator-controller-manager-58fc56757c-fg7b4\" (UID: \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:19.461588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.461541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9c42c63-830b-4588-8e94-3041c136f75d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-jrxhw\" (UID: \"a9c42c63-830b-4588-8e94-3041c136f75d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:19.461588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.461581    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qx6\" (UniqueName: \"kubernetes.io/projected/47dc36c0-8bcf-46cc-935d-ea4749bcc9d1-kube-api-access-n2qx6\") pod \"limitador-operator-controller-manager-6c549b6d85-bj8kz\" (UID: \"47dc36c0-8bcf-46cc-935d-ea4749bcc9d1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"
Jul 31 14:25:19.461697 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.461618    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-fg7b4\" (UID: \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:19.461697 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.461662    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjzs8\" (UniqueName: \"kubernetes.io/projected/a9c42c63-830b-4588-8e94-3041c136f75d-kube-api-access-fjzs8\") pod \"kuadrant-operator-controller-manager-96db98c47-jrxhw\" (UID: \"a9c42c63-830b-4588-8e94-3041c136f75d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:19.562880 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.562731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9c42c63-830b-4588-8e94-3041c136f75d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-jrxhw\" (UID: \"a9c42c63-830b-4588-8e94-3041c136f75d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:19.562880 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.562791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qx6\" (UniqueName: \"kubernetes.io/projected/47dc36c0-8bcf-46cc-935d-ea4749bcc9d1-kube-api-access-n2qx6\") pod \"limitador-operator-controller-manager-6c549b6d85-bj8kz\" (UID: \"47dc36c0-8bcf-46cc-935d-ea4749bcc9d1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"
Jul 31 14:25:19.562880 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.562830    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-fg7b4\" (UID: \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:19.562880 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.562879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjzs8\" (UniqueName: \"kubernetes.io/projected/a9c42c63-830b-4588-8e94-3041c136f75d-kube-api-access-fjzs8\") pod \"kuadrant-operator-controller-manager-96db98c47-jrxhw\" (UID: \"a9c42c63-830b-4588-8e94-3041c136f75d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:19.563945 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.562947    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wtjn\" (UniqueName: \"kubernetes.io/projected/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-kube-api-access-6wtjn\") pod \"kuadrant-operator-controller-manager-58fc56757c-fg7b4\" (UID: \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:19.563945 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.563304    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9c42c63-830b-4588-8e94-3041c136f75d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-jrxhw\" (UID: \"a9c42c63-830b-4588-8e94-3041c136f75d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:19.563945 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.563881    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-fg7b4\" (UID: \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:19.574142 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.574115    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wtjn\" (UniqueName: \"kubernetes.io/projected/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-kube-api-access-6wtjn\") pod \"kuadrant-operator-controller-manager-58fc56757c-fg7b4\" (UID: \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:19.574142 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.574134    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qx6\" (UniqueName: \"kubernetes.io/projected/47dc36c0-8bcf-46cc-935d-ea4749bcc9d1-kube-api-access-n2qx6\") pod \"limitador-operator-controller-manager-6c549b6d85-bj8kz\" (UID: \"47dc36c0-8bcf-46cc-935d-ea4749bcc9d1\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"
Jul 31 14:25:19.574298 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.574179    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjzs8\" (UniqueName: \"kubernetes.io/projected/a9c42c63-830b-4588-8e94-3041c136f75d-kube-api-access-fjzs8\") pod \"kuadrant-operator-controller-manager-96db98c47-jrxhw\" (UID: \"a9c42c63-830b-4588-8e94-3041c136f75d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:19.583719 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:19.583682    2558 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7" container="manager"
Jul 31 14:25:19.585496 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.585473    2558 generic.go:356] "Generic (PLEG): container finished" podID="a0704714-6474-4ffe-bced-21bacf5ccce6" containerID="63afdae2fc7516a16cca0e8ebb4f77cec8bf44f6beda4531d9e58972a00636aa" exitCode=0
Jul 31 14:25:19.587431 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.587364    2558 generic.go:356] "Generic (PLEG): container finished" podID="efe7928e-4412-4c73-b10a-d6052ddbb8ba" containerID="10ba55c29a6396c00f28a0a00d43fc26a3182718f0c454355780ae9f1a0495c1" exitCode=0
Jul 31 14:25:19.613860 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.613839    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:19.623840 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.623821    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:25:19.664287 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.664261    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7xmc\" (UniqueName: \"kubernetes.io/projected/efe7928e-4412-4c73-b10a-d6052ddbb8ba-kube-api-access-x7xmc\") pod \"efe7928e-4412-4c73-b10a-d6052ddbb8ba\" (UID: \"efe7928e-4412-4c73-b10a-d6052ddbb8ba\") "
Jul 31 14:25:19.664425 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.664298    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0704714-6474-4ffe-bced-21bacf5ccce6-extensions-socket-volume\") pod \"a0704714-6474-4ffe-bced-21bacf5ccce6\" (UID: \"a0704714-6474-4ffe-bced-21bacf5ccce6\") "
Jul 31 14:25:19.664425 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.664416    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhsw9\" (UniqueName: \"kubernetes.io/projected/a0704714-6474-4ffe-bced-21bacf5ccce6-kube-api-access-jhsw9\") pod \"a0704714-6474-4ffe-bced-21bacf5ccce6\" (UID: \"a0704714-6474-4ffe-bced-21bacf5ccce6\") "
Jul 31 14:25:19.664786 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.664762    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0704714-6474-4ffe-bced-21bacf5ccce6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a0704714-6474-4ffe-bced-21bacf5ccce6" (UID: "a0704714-6474-4ffe-bced-21bacf5ccce6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:25:19.666443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.666423    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0704714-6474-4ffe-bced-21bacf5ccce6-kube-api-access-jhsw9" (OuterVolumeSpecName: "kube-api-access-jhsw9") pod "a0704714-6474-4ffe-bced-21bacf5ccce6" (UID: "a0704714-6474-4ffe-bced-21bacf5ccce6"). InnerVolumeSpecName "kube-api-access-jhsw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:25:19.666650 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.666635    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe7928e-4412-4c73-b10a-d6052ddbb8ba-kube-api-access-x7xmc" (OuterVolumeSpecName: "kube-api-access-x7xmc") pod "efe7928e-4412-4c73-b10a-d6052ddbb8ba" (UID: "efe7928e-4412-4c73-b10a-d6052ddbb8ba"). InnerVolumeSpecName "kube-api-access-x7xmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:25:19.765153 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.765125    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhsw9\" (UniqueName: \"kubernetes.io/projected/a0704714-6474-4ffe-bced-21bacf5ccce6-kube-api-access-jhsw9\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:19.765153 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.765150    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7xmc\" (UniqueName: \"kubernetes.io/projected/efe7928e-4412-4c73-b10a-d6052ddbb8ba-kube-api-access-x7xmc\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:19.765153 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.765160    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0704714-6474-4ffe-bced-21bacf5ccce6-extensions-socket-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:19.770875 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.770845    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:19.777801 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.777781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:19.784656 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.784630    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"
Jul 31 14:25:19.936471 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:19.936428    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"]
Jul 31 14:25:19.939235 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:25:19.939209    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d19754e_90ef_40f6_a0f4_95e0a6a8fe40.slice/crio-bb9f1a050142fc2717b0ea0ea71bbb3f1285fc1b12348141bcc29187b69de66f WatchSource:0}: Error finding container bb9f1a050142fc2717b0ea0ea71bbb3f1285fc1b12348141bcc29187b69de66f: Status 404 returned error can't find the container with id bb9f1a050142fc2717b0ea0ea71bbb3f1285fc1b12348141bcc29187b69de66f
Jul 31 14:25:20.157542 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.157515    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"]
Jul 31 14:25:20.159550 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:25:20.159513    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dc36c0_8bcf_46cc_935d_ea4749bcc9d1.slice/crio-04b3761a981b072bbbef45ab72a7665cbddeef8c3ad9ad3197eaa715a2031bb6 WatchSource:0}: Error finding container 04b3761a981b072bbbef45ab72a7665cbddeef8c3ad9ad3197eaa715a2031bb6: Status 404 returned error can't find the container with id 04b3761a981b072bbbef45ab72a7665cbddeef8c3ad9ad3197eaa715a2031bb6
Jul 31 14:25:20.161087 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.161066    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"]
Jul 31 14:25:20.162875 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:25:20.162856    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c42c63_830b_4588_8e94_3041c136f75d.slice/crio-7e4d37d7d10007f66849b936d3779038cac91ec52d2e8a314b176ad6668b5334 WatchSource:0}: Error finding container 7e4d37d7d10007f66849b936d3779038cac91ec52d2e8a314b176ad6668b5334: Status 404 returned error can't find the container with id 7e4d37d7d10007f66849b936d3779038cac91ec52d2e8a314b176ad6668b5334
Jul 31 14:25:20.594372 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.594345    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7jtlp"
Jul 31 14:25:20.594821 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.594394    2558 scope.go:117] "RemoveContainer" containerID="10ba55c29a6396c00f28a0a00d43fc26a3182718f0c454355780ae9f1a0495c1"
Jul 31 14:25:20.596322 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.596291    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4" event={"ID":"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40","Type":"ContainerStarted","Data":"41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac"}
Jul 31 14:25:20.596443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.596339    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4" event={"ID":"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40","Type":"ContainerStarted","Data":"bb9f1a050142fc2717b0ea0ea71bbb3f1285fc1b12348141bcc29187b69de66f"}
Jul 31 14:25:20.596443 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.596383    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:20.598170 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.598139    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw" event={"ID":"a9c42c63-830b-4588-8e94-3041c136f75d","Type":"ContainerStarted","Data":"61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb"}
Jul 31 14:25:20.598447 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.598192    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw" event={"ID":"a9c42c63-830b-4588-8e94-3041c136f75d","Type":"ContainerStarted","Data":"7e4d37d7d10007f66849b936d3779038cac91ec52d2e8a314b176ad6668b5334"}
Jul 31 14:25:20.598447 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.598221    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:20.599833 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.599810    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" containerName="manager" containerID="cri-o://44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae" gracePeriod=2
Jul 31 14:25:20.601323 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.601303    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-vbhp7"
Jul 31 14:25:20.603344 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.603319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz" event={"ID":"47dc36c0-8bcf-46cc-935d-ea4749bcc9d1","Type":"ContainerStarted","Data":"c33533910fbb2cf6b9b80d6023c345e2e99fedd221a715bbbb88e91b8274b80f"}
Jul 31 14:25:20.603446 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.603364    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz" event={"ID":"47dc36c0-8bcf-46cc-935d-ea4749bcc9d1","Type":"ContainerStarted","Data":"04b3761a981b072bbbef45ab72a7665cbddeef8c3ad9ad3197eaa715a2031bb6"}
Jul 31 14:25:20.603829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.603808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"
Jul 31 14:25:20.604739 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.604721    2558 scope.go:117] "RemoveContainer" containerID="63afdae2fc7516a16cca0e8ebb4f77cec8bf44f6beda4531d9e58972a00636aa"
Jul 31 14:25:20.624863 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.624818    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4" podStartSLOduration=1.62480269 podStartE2EDuration="1.62480269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:25:20.620936883 +0000 UTC m=+759.493500142" watchObservedRunningTime="2026-07-31 14:25:20.62480269 +0000 UTC m=+759.497366025"
Jul 31 14:25:20.648691 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.648643    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz" podStartSLOduration=1.6486295420000001 podStartE2EDuration="1.648629542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:25:20.647888843 +0000 UTC m=+759.520452098" watchObservedRunningTime="2026-07-31 14:25:20.648629542 +0000 UTC m=+759.521192795"
Jul 31 14:25:20.650082 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:20.650052    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:20.677695 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.677639    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw" podStartSLOduration=1.677619291 podStartE2EDuration="1.677619291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:25:20.67514849 +0000 UTC m=+759.547711744" watchObservedRunningTime="2026-07-31 14:25:20.677619291 +0000 UTC m=+759.550182545"
Jul 31 14:25:20.851632 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.851574    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:20.853815 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:20.853787    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:20.874238 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.874217    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lntcw\" (UniqueName: \"kubernetes.io/projected/795f000a-48c8-4067-89f9-7e22eab913d3-kube-api-access-lntcw\") pod \"795f000a-48c8-4067-89f9-7e22eab913d3\" (UID: \"795f000a-48c8-4067-89f9-7e22eab913d3\") "
Jul 31 14:25:20.874428 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.874410    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f000a-48c8-4067-89f9-7e22eab913d3-extensions-socket-volume\") pod \"795f000a-48c8-4067-89f9-7e22eab913d3\" (UID: \"795f000a-48c8-4067-89f9-7e22eab913d3\") "
Jul 31 14:25:20.874768 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.874745    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795f000a-48c8-4067-89f9-7e22eab913d3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "795f000a-48c8-4067-89f9-7e22eab913d3" (UID: "795f000a-48c8-4067-89f9-7e22eab913d3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:25:20.876255 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.876234    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795f000a-48c8-4067-89f9-7e22eab913d3-kube-api-access-lntcw" (OuterVolumeSpecName: "kube-api-access-lntcw") pod "795f000a-48c8-4067-89f9-7e22eab913d3" (UID: "795f000a-48c8-4067-89f9-7e22eab913d3"). InnerVolumeSpecName "kube-api-access-lntcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:25:20.975792 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.975762    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lntcw\" (UniqueName: \"kubernetes.io/projected/795f000a-48c8-4067-89f9-7e22eab913d3-kube-api-access-lntcw\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:20.975792 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:20.975787    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f000a-48c8-4067-89f9-7e22eab913d3-extensions-socket-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:21.577488 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:21.577449    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:21.579569 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:21.579541    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" path="/var/lib/kubelet/pods/795f000a-48c8-4067-89f9-7e22eab913d3/volumes"
Jul 31 14:25:21.580001 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:21.579982    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0704714-6474-4ffe-bced-21bacf5ccce6" path="/var/lib/kubelet/pods/a0704714-6474-4ffe-bced-21bacf5ccce6/volumes"
Jul 31 14:25:21.580344 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:21.580328    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe7928e-4412-4c73-b10a-d6052ddbb8ba" path="/var/lib/kubelet/pods/efe7928e-4412-4c73-b10a-d6052ddbb8ba/volumes"
Jul 31 14:25:21.609280 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:21.609248    2558 generic.go:356] "Generic (PLEG): container finished" podID="795f000a-48c8-4067-89f9-7e22eab913d3" containerID="44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae" exitCode=0
Jul 31 14:25:21.609662 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:21.609299    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:21.609662 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:21.609317    2558 scope.go:117] "RemoveContainer" containerID="44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae"
Jul 31 14:25:21.614285 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:21.614260    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-h6mg7\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="795f000a-48c8-4067-89f9-7e22eab913d3" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-h6mg7"
Jul 31 14:25:21.618154 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:21.618130    2558 scope.go:117] "RemoveContainer" containerID="44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae"
Jul 31 14:25:21.618386 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:21.618367    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae\": container with ID starting with 44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae not found: ID does not exist" containerID="44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae"
Jul 31 14:25:21.618433 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:21.618394    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae"} err="failed to get container status \"44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae\": rpc error: code = NotFound desc = could not find container \"44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae\": container with ID starting with 44e86ac1ae3f968c96c6f70479d2d16c76480aedf37268bd1d1392f6e66a39ae not found: ID does not exist"
Jul 31 14:25:31.614796 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.614765    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:31.615292 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.615182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:31.696943 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.696908    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"]
Jul 31 14:25:31.697166 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.697138    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4" podUID="3d19754e-90ef-40f6-a0f4-95e0a6a8fe40" containerName="manager" containerID="cri-o://41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac" gracePeriod=10
Jul 31 14:25:31.938806 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.938784    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:31.975601 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.975579    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wtjn\" (UniqueName: \"kubernetes.io/projected/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-kube-api-access-6wtjn\") pod \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\" (UID: \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\") "
Jul 31 14:25:31.975765 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.975645    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-extensions-socket-volume\") pod \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\" (UID: \"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40\") "
Jul 31 14:25:31.975979 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.975956    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3d19754e-90ef-40f6-a0f4-95e0a6a8fe40" (UID: "3d19754e-90ef-40f6-a0f4-95e0a6a8fe40"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:25:31.977974 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.977942    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-kube-api-access-6wtjn" (OuterVolumeSpecName: "kube-api-access-6wtjn") pod "3d19754e-90ef-40f6-a0f4-95e0a6a8fe40" (UID: "3d19754e-90ef-40f6-a0f4-95e0a6a8fe40"). InnerVolumeSpecName "kube-api-access-6wtjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:25:31.988220 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.988196    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"]
Jul 31 14:25:31.988828 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.988803    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d19754e-90ef-40f6-a0f4-95e0a6a8fe40" containerName="manager"
Jul 31 14:25:31.988828 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.988829    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d19754e-90ef-40f6-a0f4-95e0a6a8fe40" containerName="manager"
Jul 31 14:25:31.988990 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.988964    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d19754e-90ef-40f6-a0f4-95e0a6a8fe40" containerName="manager"
Jul 31 14:25:31.997337 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:31.997313    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.004762 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.004734    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"]
Jul 31 14:25:32.076226 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.076193    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9pgh\" (UniqueName: \"kubernetes.io/projected/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-kube-api-access-l9pgh\") pod \"kuadrant-operator-controller-manager-68cf589589-ltfr8\" (UID: \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.076226 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.076236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-ltfr8\" (UID: \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.076436 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.076352    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wtjn\" (UniqueName: \"kubernetes.io/projected/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-kube-api-access-6wtjn\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:32.076436 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.076362    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40-extensions-socket-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:32.177603 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.177522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-ltfr8\" (UID: \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.177742 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.177634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9pgh\" (UniqueName: \"kubernetes.io/projected/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-kube-api-access-l9pgh\") pod \"kuadrant-operator-controller-manager-68cf589589-ltfr8\" (UID: \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.177988 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.177965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-ltfr8\" (UID: \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.188634 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.188607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9pgh\" (UniqueName: \"kubernetes.io/projected/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-kube-api-access-l9pgh\") pod \"kuadrant-operator-controller-manager-68cf589589-ltfr8\" (UID: \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.309775 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.309742    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.447484 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.447456    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"]
Jul 31 14:25:32.449482 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:25:32.449460    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7672885_b77f_4580_ad9c_2dfcfe1e94e6.slice/crio-4bf369f952136a98373b53736edbf65fa2edb531e297a5018f5108e726e98ceb WatchSource:0}: Error finding container 4bf369f952136a98373b53736edbf65fa2edb531e297a5018f5108e726e98ceb: Status 404 returned error can't find the container with id 4bf369f952136a98373b53736edbf65fa2edb531e297a5018f5108e726e98ceb
Jul 31 14:25:32.621052 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.621000    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-bj8kz"
Jul 31 14:25:32.674103 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.674065    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8" event={"ID":"e7672885-b77f-4580-ad9c-2dfcfe1e94e6","Type":"ContainerStarted","Data":"48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f"}
Jul 31 14:25:32.674265 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.674140    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:32.674265 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.674166    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8" event={"ID":"e7672885-b77f-4580-ad9c-2dfcfe1e94e6","Type":"ContainerStarted","Data":"4bf369f952136a98373b53736edbf65fa2edb531e297a5018f5108e726e98ceb"}
Jul 31 14:25:32.675624 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.675596    2558 generic.go:356] "Generic (PLEG): container finished" podID="3d19754e-90ef-40f6-a0f4-95e0a6a8fe40" containerID="41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac" exitCode=0
Jul 31 14:25:32.675767 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.675649    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"
Jul 31 14:25:32.675767 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.675681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4" event={"ID":"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40","Type":"ContainerDied","Data":"41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac"}
Jul 31 14:25:32.675767 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.675735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4" event={"ID":"3d19754e-90ef-40f6-a0f4-95e0a6a8fe40","Type":"ContainerDied","Data":"bb9f1a050142fc2717b0ea0ea71bbb3f1285fc1b12348141bcc29187b69de66f"}
Jul 31 14:25:32.675767 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.675760    2558 scope.go:117] "RemoveContainer" containerID="41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac"
Jul 31 14:25:32.684475 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.684458    2558 scope.go:117] "RemoveContainer" containerID="41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac"
Jul 31 14:25:32.684743 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:32.684718    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac\": container with ID starting with 41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac not found: ID does not exist" containerID="41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac"
Jul 31 14:25:32.684835 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.684754    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac"} err="failed to get container status \"41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac\": rpc error: code = NotFound desc = could not find container \"41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac\": container with ID starting with 41e9cd33330675d22c6e5a89bd2404e08651cb7dd038a9bf3c33c58f3fb24aac not found: ID does not exist"
Jul 31 14:25:32.695360 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.695317    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8" podStartSLOduration=1.695302496 podStartE2EDuration="1.695302496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:25:32.69382724 +0000 UTC m=+771.566390493" watchObservedRunningTime="2026-07-31 14:25:32.695302496 +0000 UTC m=+771.567865749"
Jul 31 14:25:32.716537 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.716470    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"]
Jul 31 14:25:32.721536 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:32.721512    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-fg7b4"]
Jul 31 14:25:33.580420 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:33.580387    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d19754e-90ef-40f6-a0f4-95e0a6a8fe40" path="/var/lib/kubelet/pods/3d19754e-90ef-40f6-a0f4-95e0a6a8fe40/volumes"
Jul 31 14:25:43.684415 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:43.684386    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:25:43.735937 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:43.735905    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"]
Jul 31 14:25:43.736211 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:43.736161    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw" podUID="a9c42c63-830b-4588-8e94-3041c136f75d" containerName="manager" containerID="cri-o://61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb" gracePeriod=10
Jul 31 14:25:43.988186 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:43.988160    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:44.089159 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.089125    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9c42c63-830b-4588-8e94-3041c136f75d-extensions-socket-volume\") pod \"a9c42c63-830b-4588-8e94-3041c136f75d\" (UID: \"a9c42c63-830b-4588-8e94-3041c136f75d\") "
Jul 31 14:25:44.089337 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.089233    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjzs8\" (UniqueName: \"kubernetes.io/projected/a9c42c63-830b-4588-8e94-3041c136f75d-kube-api-access-fjzs8\") pod \"a9c42c63-830b-4588-8e94-3041c136f75d\" (UID: \"a9c42c63-830b-4588-8e94-3041c136f75d\") "
Jul 31 14:25:44.089584 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.089554    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c42c63-830b-4588-8e94-3041c136f75d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a9c42c63-830b-4588-8e94-3041c136f75d" (UID: "a9c42c63-830b-4588-8e94-3041c136f75d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:25:44.091263 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.091243    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c42c63-830b-4588-8e94-3041c136f75d-kube-api-access-fjzs8" (OuterVolumeSpecName: "kube-api-access-fjzs8") pod "a9c42c63-830b-4588-8e94-3041c136f75d" (UID: "a9c42c63-830b-4588-8e94-3041c136f75d"). InnerVolumeSpecName "kube-api-access-fjzs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:25:44.190545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.190515    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjzs8\" (UniqueName: \"kubernetes.io/projected/a9c42c63-830b-4588-8e94-3041c136f75d-kube-api-access-fjzs8\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:44.190545 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.190541    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9c42c63-830b-4588-8e94-3041c136f75d-extensions-socket-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:25:44.745406 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.745366    2558 generic.go:356] "Generic (PLEG): container finished" podID="a9c42c63-830b-4588-8e94-3041c136f75d" containerID="61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb" exitCode=0
Jul 31 14:25:44.745867 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.745415    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"
Jul 31 14:25:44.745867 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.745456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw" event={"ID":"a9c42c63-830b-4588-8e94-3041c136f75d","Type":"ContainerDied","Data":"61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb"}
Jul 31 14:25:44.745867 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.745512    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw" event={"ID":"a9c42c63-830b-4588-8e94-3041c136f75d","Type":"ContainerDied","Data":"7e4d37d7d10007f66849b936d3779038cac91ec52d2e8a314b176ad6668b5334"}
Jul 31 14:25:44.745867 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.745534    2558 scope.go:117] "RemoveContainer" containerID="61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb"
Jul 31 14:25:44.754736 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.754716    2558 scope.go:117] "RemoveContainer" containerID="61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb"
Jul 31 14:25:44.754989 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:25:44.754963    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb\": container with ID starting with 61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb not found: ID does not exist" containerID="61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb"
Jul 31 14:25:44.755076 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.755003    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb"} err="failed to get container status \"61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb\": rpc error: code = NotFound desc = could not find container \"61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb\": container with ID starting with 61b766c6b3e70609e5c4d6624f62dcf60f29401fe17bfadf5ed6fa5c1d2b58eb not found: ID does not exist"
Jul 31 14:25:44.767698 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.767673    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"]
Jul 31 14:25:44.770584 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:44.770561    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-jrxhw"]
Jul 31 14:25:45.579918 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:45.579842    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9c42c63-830b-4588-8e94-3041c136f75d" path="/var/lib/kubelet/pods/a9c42c63-830b-4588-8e94-3041c136f75d/volumes"
Jul 31 14:25:48.062630 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.062589    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"]
Jul 31 14:25:48.063254 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.063233    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c42c63-830b-4588-8e94-3041c136f75d" containerName="manager"
Jul 31 14:25:48.063254 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.063253    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c42c63-830b-4588-8e94-3041c136f75d" containerName="manager"
Jul 31 14:25:48.063585 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.063560    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c42c63-830b-4588-8e94-3041c136f75d" containerName="manager"
Jul 31 14:25:48.066992 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.066974    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.069594 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.069569    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-sxm6m\""
Jul 31 14:25:48.077380 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.077359    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"]
Jul 31 14:25:48.225918 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.225886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.225918 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.225920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.226206 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.225971    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.226206 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.226084    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.226206 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.226123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.226206 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.226199    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.226409 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.226231    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.226409 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.226270    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwxk\" (UniqueName: \"kubernetes.io/projected/7c5561a1-775e-4ccb-a096-19bbd398bde0-kube-api-access-6pwxk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.226409 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.226306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5561a1-775e-4ccb-a096-19bbd398bde0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.327674 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.327595    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.327674 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.327631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.327674 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.327652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwxk\" (UniqueName: \"kubernetes.io/projected/7c5561a1-775e-4ccb-a096-19bbd398bde0-kube-api-access-6pwxk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.327942 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.327841    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5561a1-775e-4ccb-a096-19bbd398bde0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.327942 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.327918    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.327942 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.327938    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.328139 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.327975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.328139 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.328017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.328139 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.328070    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.328348 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.328329    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.328446 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.328421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.328493 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.328439    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.328566 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.328499    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.328613 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.328600    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c5561a1-775e-4ccb-a096-19bbd398bde0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.330150 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.330130    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.330426 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.330410    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.335500 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.335473    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c5561a1-775e-4ccb-a096-19bbd398bde0-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.335702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.335685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwxk\" (UniqueName: \"kubernetes.io/projected/7c5561a1-775e-4ccb-a096-19bbd398bde0-kube-api-access-6pwxk\") pod \"maas-default-gateway-openshift-default-5ccb968c66-x79gg\" (UID: \"7c5561a1-775e-4ccb-a096-19bbd398bde0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.378380 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.378354    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:48.509242 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.509217    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"]
Jul 31 14:25:48.511122 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:25:48.511097    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5561a1_775e_4ccb_a096_19bbd398bde0.slice/crio-5600201d536fd1d9ba4b269f3918a05b3124261dde9090a4dfeceaecfd7c29e1 WatchSource:0}: Error finding container 5600201d536fd1d9ba4b269f3918a05b3124261dde9090a4dfeceaecfd7c29e1: Status 404 returned error can't find the container with id 5600201d536fd1d9ba4b269f3918a05b3124261dde9090a4dfeceaecfd7c29e1
Jul 31 14:25:48.513307 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.513265    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:25:48.513400 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.513362    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:25:48.513451 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.513405    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:25:48.770964 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.770874    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" event={"ID":"7c5561a1-775e-4ccb-a096-19bbd398bde0","Type":"ContainerStarted","Data":"7a06779a857445d5e1ce0e5a3f47edb09a490a23c88a80d50af6ea0ae8edda00"}
Jul 31 14:25:48.770964 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.770947    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" event={"ID":"7c5561a1-775e-4ccb-a096-19bbd398bde0","Type":"ContainerStarted","Data":"5600201d536fd1d9ba4b269f3918a05b3124261dde9090a4dfeceaecfd7c29e1"}
Jul 31 14:25:48.789693 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:48.789637    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" podStartSLOduration=0.789616118 podStartE2EDuration="789.616118ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:25:48.789405817 +0000 UTC m=+787.661969071" watchObservedRunningTime="2026-07-31 14:25:48.789616118 +0000 UTC m=+787.662179372"
Jul 31 14:25:49.379573 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:49.379541    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:49.384074 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:49.384050    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:49.776172 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:49.776097    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:25:49.777037 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:25:49.777007    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:26:00.867313 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:00.867281    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-987b4"]
Jul 31 14:26:00.870584 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:00.870561    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-987b4"
Jul 31 14:26:00.873226 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:00.873210    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wx7kw\""
Jul 31 14:26:00.877590 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:00.877566    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-987b4"]
Jul 31 14:26:00.948292 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:00.948258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjbr\" (UniqueName: \"kubernetes.io/projected/99c380bf-13af-4c78-8112-ae073a6915eb-kube-api-access-mcjbr\") pod \"authorino-b54df94b4-987b4\" (UID: \"99c380bf-13af-4c78-8112-ae073a6915eb\") " pod="kuadrant-system/authorino-b54df94b4-987b4"
Jul 31 14:26:01.049424 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:01.049390    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjbr\" (UniqueName: \"kubernetes.io/projected/99c380bf-13af-4c78-8112-ae073a6915eb-kube-api-access-mcjbr\") pod \"authorino-b54df94b4-987b4\" (UID: \"99c380bf-13af-4c78-8112-ae073a6915eb\") " pod="kuadrant-system/authorino-b54df94b4-987b4"
Jul 31 14:26:01.057332 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:01.057305    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjbr\" (UniqueName: \"kubernetes.io/projected/99c380bf-13af-4c78-8112-ae073a6915eb-kube-api-access-mcjbr\") pod \"authorino-b54df94b4-987b4\" (UID: \"99c380bf-13af-4c78-8112-ae073a6915eb\") " pod="kuadrant-system/authorino-b54df94b4-987b4"
Jul 31 14:26:01.181570 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:01.181490    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-987b4"
Jul 31 14:26:01.314472 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:01.314449    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-987b4"]
Jul 31 14:26:01.316103 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:26:01.316076    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99c380bf_13af_4c78_8112_ae073a6915eb.slice/crio-313314c180f6422166a2d14cd2fc42accf5a2d56753372502510a76d3c34a0c8 WatchSource:0}: Error finding container 313314c180f6422166a2d14cd2fc42accf5a2d56753372502510a76d3c34a0c8: Status 404 returned error can't find the container with id 313314c180f6422166a2d14cd2fc42accf5a2d56753372502510a76d3c34a0c8
Jul 31 14:26:01.834788 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:01.834741    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-987b4" event={"ID":"99c380bf-13af-4c78-8112-ae073a6915eb","Type":"ContainerStarted","Data":"313314c180f6422166a2d14cd2fc42accf5a2d56753372502510a76d3c34a0c8"}
Jul 31 14:26:03.849962 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:03.849916    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-987b4" event={"ID":"99c380bf-13af-4c78-8112-ae073a6915eb","Type":"ContainerStarted","Data":"58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13"}
Jul 31 14:26:03.865272 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:03.865223    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-987b4" podStartSLOduration=1.450751161 podStartE2EDuration="3.865207673s" totalImagesPullingTime="2.414456512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:26:03.864849118 +0000 UTC m=+802.737412368" watchObservedRunningTime="2026-07-31 14:26:03.865207673 +0000 UTC m=+802.737770925"
Jul 31 14:26:06.196917 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.196873    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-987b4"]
Jul 31 14:26:06.197372 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.197137    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-987b4" podUID="99c380bf-13af-4c78-8112-ae073a6915eb" containerName="authorino" containerID="cri-o://58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13" gracePeriod=30
Jul 31 14:26:06.448599 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.448543    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-987b4"
Jul 31 14:26:06.497264 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.497236    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcjbr\" (UniqueName: \"kubernetes.io/projected/99c380bf-13af-4c78-8112-ae073a6915eb-kube-api-access-mcjbr\") pod \"99c380bf-13af-4c78-8112-ae073a6915eb\" (UID: \"99c380bf-13af-4c78-8112-ae073a6915eb\") "
Jul 31 14:26:06.499261 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.499231    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c380bf-13af-4c78-8112-ae073a6915eb-kube-api-access-mcjbr" (OuterVolumeSpecName: "kube-api-access-mcjbr") pod "99c380bf-13af-4c78-8112-ae073a6915eb" (UID: "99c380bf-13af-4c78-8112-ae073a6915eb"). InnerVolumeSpecName "kube-api-access-mcjbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:26:06.598713 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.598682    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcjbr\" (UniqueName: \"kubernetes.io/projected/99c380bf-13af-4c78-8112-ae073a6915eb-kube-api-access-mcjbr\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:26:06.866739 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.866702    2558 generic.go:356] "Generic (PLEG): container finished" podID="99c380bf-13af-4c78-8112-ae073a6915eb" containerID="58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13" exitCode=0
Jul 31 14:26:06.866908 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.866748    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-987b4"
Jul 31 14:26:06.866908 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.866785    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-987b4" event={"ID":"99c380bf-13af-4c78-8112-ae073a6915eb","Type":"ContainerDied","Data":"58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13"}
Jul 31 14:26:06.866908 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.866830    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-987b4" event={"ID":"99c380bf-13af-4c78-8112-ae073a6915eb","Type":"ContainerDied","Data":"313314c180f6422166a2d14cd2fc42accf5a2d56753372502510a76d3c34a0c8"}
Jul 31 14:26:06.866908 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.866847    2558 scope.go:117] "RemoveContainer" containerID="58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13"
Jul 31 14:26:06.876218 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.876199    2558 scope.go:117] "RemoveContainer" containerID="58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13"
Jul 31 14:26:06.876459 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:26:06.876439    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13\": container with ID starting with 58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13 not found: ID does not exist" containerID="58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13"
Jul 31 14:26:06.876529 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.876472    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13"} err="failed to get container status \"58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13\": rpc error: code = NotFound desc = could not find container \"58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13\": container with ID starting with 58be134a9fd60d36d27f3c21feb29215635d0d2f2541367b2d5c9af23839fb13 not found: ID does not exist"
Jul 31 14:26:06.887993 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.887967    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-987b4"]
Jul 31 14:26:06.890116 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:06.890094    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-987b4"]
Jul 31 14:26:07.580739 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:07.580705    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c380bf-13af-4c78-8112-ae073a6915eb" path="/var/lib/kubelet/pods/99c380bf-13af-4c78-8112-ae073a6915eb/volumes"
Jul 31 14:26:21.729765 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.729733    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"]
Jul 31 14:26:21.730336 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.730324    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c380bf-13af-4c78-8112-ae073a6915eb" containerName="authorino"
Jul 31 14:26:21.730416 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.730341    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c380bf-13af-4c78-8112-ae073a6915eb" containerName="authorino"
Jul 31 14:26:21.730468 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.730433    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c380bf-13af-4c78-8112-ae073a6915eb" containerName="authorino"
Jul 31 14:26:21.736126 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.736098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:26:21.740423 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.740369    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 31 14:26:21.740423 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.740368    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 31 14:26:21.740678 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.740656    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 31 14:26:21.740831 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.740814    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-m7js5\""
Jul 31 14:26:21.741806 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.741788    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"]
Jul 31 14:26:21.841080 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.841012    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2763d69b-797f-4001-be9a-bb56210c869a-data\") pod \"postgres-69fb4594cf-kv5tz\" (UID: \"2763d69b-797f-4001-be9a-bb56210c869a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:26:21.841264 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.841131    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pmj\" (UniqueName: \"kubernetes.io/projected/2763d69b-797f-4001-be9a-bb56210c869a-kube-api-access-26pmj\") pod \"postgres-69fb4594cf-kv5tz\" (UID: \"2763d69b-797f-4001-be9a-bb56210c869a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:26:21.942233 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.942197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pmj\" (UniqueName: \"kubernetes.io/projected/2763d69b-797f-4001-be9a-bb56210c869a-kube-api-access-26pmj\") pod \"postgres-69fb4594cf-kv5tz\" (UID: \"2763d69b-797f-4001-be9a-bb56210c869a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:26:21.942441 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.942332    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2763d69b-797f-4001-be9a-bb56210c869a-data\") pod \"postgres-69fb4594cf-kv5tz\" (UID: \"2763d69b-797f-4001-be9a-bb56210c869a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:26:21.942709 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.942689    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/2763d69b-797f-4001-be9a-bb56210c869a-data\") pod \"postgres-69fb4594cf-kv5tz\" (UID: \"2763d69b-797f-4001-be9a-bb56210c869a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:26:21.950733 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:21.950708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pmj\" (UniqueName: \"kubernetes.io/projected/2763d69b-797f-4001-be9a-bb56210c869a-kube-api-access-26pmj\") pod \"postgres-69fb4594cf-kv5tz\" (UID: \"2763d69b-797f-4001-be9a-bb56210c869a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:26:22.048217 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:22.048192    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:26:22.183846 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:22.183823    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"]
Jul 31 14:26:22.185435 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:26:22.185395    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2763d69b_797f_4001_be9a_bb56210c869a.slice/crio-1dafb4c63fc409294d175a8c282da1aacee3e59f42c3a3fcbdaf718667af82d7 WatchSource:0}: Error finding container 1dafb4c63fc409294d175a8c282da1aacee3e59f42c3a3fcbdaf718667af82d7: Status 404 returned error can't find the container with id 1dafb4c63fc409294d175a8c282da1aacee3e59f42c3a3fcbdaf718667af82d7
Jul 31 14:26:22.955882 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:26:22.955849    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz" event={"ID":"2763d69b-797f-4001-be9a-bb56210c869a","Type":"ContainerStarted","Data":"1dafb4c63fc409294d175a8c282da1aacee3e59f42c3a3fcbdaf718667af82d7"}
Jul 31 14:26:32.466499 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:26:32.466451    2558 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/rhel9/postgresql-16:latest: reading manifest sha256:1ecfd9f37f53a07778d1d2073737e2dfa298fe7c62cd637f111ec30680186f1a in registry.redhat.io/rhel9/postgresql-16: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="registry.redhat.io/rhel9/postgresql-16:latest"
Jul 31 14:26:32.466906 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:26:32.466576    2558 kuberuntime_manager.go:1459] "Unhandled Error" err="container postgres start failed in pod postgres-69fb4594cf-kv5tz_odh-ai-gateway-infra(2763d69b-797f-4001-be9a-bb56210c869a): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/rhel9/postgresql-16:latest: reading manifest sha256:1ecfd9f37f53a07778d1d2073737e2dfa298fe7c62cd637f111ec30680186f1a in registry.redhat.io/rhel9/postgresql-16: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 31 14:26:32.466906 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:26:32.466609    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"postgres\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/rhel9/postgresql-16:latest: reading manifest sha256:1ecfd9f37f53a07778d1d2073737e2dfa298fe7c62cd637f111ec30680186f1a in registry.redhat.io/rhel9/postgresql-16: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz" podUID="2763d69b-797f-4001-be9a-bb56210c869a"
Jul 31 14:26:33.009340 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:26:33.009285    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"postgres\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/rhel9/postgresql-16:latest\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://registry.redhat.io/rhel9/postgresql-16:latest: reading manifest sha256:1ecfd9f37f53a07778d1d2073737e2dfa298fe7c62cd637f111ec30680186f1a in registry.redhat.io/rhel9/postgresql-16: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz" podUID="2763d69b-797f-4001-be9a-bb56210c869a"
Jul 31 14:26:54.736673 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:26:54.736623    2558 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/rhel9/postgresql-16:latest: pinging container registry registry.redhat.io: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="registry.redhat.io/rhel9/postgresql-16:latest"
Jul 31 14:26:54.737142 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:26:54.736749    2558 kuberuntime_manager.go:1459] "Unhandled Error" err="container postgres start failed in pod postgres-69fb4594cf-kv5tz_odh-ai-gateway-infra(2763d69b-797f-4001-be9a-bb56210c869a): ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/rhel9/postgresql-16:latest: pinging container registry registry.redhat.io: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 31 14:26:54.737142 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:26:54.736795    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"postgres\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/rhel9/postgresql-16:latest: pinging container registry registry.redhat.io: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz" podUID="2763d69b-797f-4001-be9a-bb56210c869a"
Jul 31 14:27:06.574381 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:27:06.574342    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"postgres\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/rhel9/postgresql-16:latest\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: initializing source docker://registry.redhat.io/rhel9/postgresql-16:latest: pinging container registry registry.redhat.io: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz" podUID="2763d69b-797f-4001-be9a-bb56210c869a"
Jul 31 14:27:24.464465 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:24.464439    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 31 14:27:25.279006 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:25.278969    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz" event={"ID":"2763d69b-797f-4001-be9a-bb56210c869a","Type":"ContainerStarted","Data":"fdb8e0cf26864637e4660fa1b7b31ed1d75c5a4a6e154b7c38fac430158c22da"}
Jul 31 14:27:25.279212 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:25.279191    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:27:25.296549 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:25.296503    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz" podStartSLOduration=2.021887426 podStartE2EDuration="1m4.296489081s" totalImagesPullingTime="1m2.274601655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-07-31 14:27:25.294590947 +0000 UTC m=+884.167154200" watchObservedRunningTime="2026-07-31 14:27:25.296489081 +0000 UTC m=+884.169052334"
Jul 31 14:27:31.312953 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:31.312920    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kv5tz"
Jul 31 14:27:32.214405 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.214369    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fb488b8bd-n26kn"]
Jul 31 14:27:32.221846 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.221797    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:27:32.224203 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.224179    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fb488b8bd-n26kn"]
Jul 31 14:27:32.225866 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.225844    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wx7kw\""
Jul 31 14:27:32.225968 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.225844    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 31 14:27:32.290462 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.290433    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sd69\" (UniqueName: \"kubernetes.io/projected/dd09525a-39b8-4cbc-a844-7a5006a33717-kube-api-access-2sd69\") pod \"authorino-6fb488b8bd-n26kn\" (UID: \"dd09525a-39b8-4cbc-a844-7a5006a33717\") " pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:27:32.290617 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.290489    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd09525a-39b8-4cbc-a844-7a5006a33717-tls-cert\") pod \"authorino-6fb488b8bd-n26kn\" (UID: \"dd09525a-39b8-4cbc-a844-7a5006a33717\") " pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:27:32.391573 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.391541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd09525a-39b8-4cbc-a844-7a5006a33717-tls-cert\") pod \"authorino-6fb488b8bd-n26kn\" (UID: \"dd09525a-39b8-4cbc-a844-7a5006a33717\") " pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:27:32.391937 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.391630    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sd69\" (UniqueName: \"kubernetes.io/projected/dd09525a-39b8-4cbc-a844-7a5006a33717-kube-api-access-2sd69\") pod \"authorino-6fb488b8bd-n26kn\" (UID: \"dd09525a-39b8-4cbc-a844-7a5006a33717\") " pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:27:32.394150 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.394129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd09525a-39b8-4cbc-a844-7a5006a33717-tls-cert\") pod \"authorino-6fb488b8bd-n26kn\" (UID: \"dd09525a-39b8-4cbc-a844-7a5006a33717\") " pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:27:32.402553 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.402532    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sd69\" (UniqueName: \"kubernetes.io/projected/dd09525a-39b8-4cbc-a844-7a5006a33717-kube-api-access-2sd69\") pod \"authorino-6fb488b8bd-n26kn\" (UID: \"dd09525a-39b8-4cbc-a844-7a5006a33717\") " pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:27:32.555556 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.555529    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:27:32.687762 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:32.687733    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fb488b8bd-n26kn"]
Jul 31 14:27:32.690177 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:27:32.690154    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd09525a_39b8_4cbc_a844_7a5006a33717.slice/crio-ebf84191688f4f2db0aabf0d216d4d971d3b8a95682bb4f9c6032dc569126d52 WatchSource:0}: Error finding container ebf84191688f4f2db0aabf0d216d4d971d3b8a95682bb4f9c6032dc569126d52: Status 404 returned error can't find the container with id ebf84191688f4f2db0aabf0d216d4d971d3b8a95682bb4f9c6032dc569126d52
Jul 31 14:27:33.323796 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:33.323722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fb488b8bd-n26kn" event={"ID":"dd09525a-39b8-4cbc-a844-7a5006a33717","Type":"ContainerStarted","Data":"ebf84191688f4f2db0aabf0d216d4d971d3b8a95682bb4f9c6032dc569126d52"}
Jul 31 14:27:34.330010 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:34.329977    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fb488b8bd-n26kn" event={"ID":"dd09525a-39b8-4cbc-a844-7a5006a33717","Type":"ContainerStarted","Data":"1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e"}
Jul 31 14:27:34.349792 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:34.349743    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6fb488b8bd-n26kn" podStartSLOduration=0.874063862 podStartE2EDuration="2.349727149s" totalImagesPullingTime="1.475663287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:27:34.346792034 +0000 UTC m=+893.219355288" watchObservedRunningTime="2026-07-31 14:27:34.349727149 +0000 UTC m=+893.222290401"
Jul 31 14:27:45.348218 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:45.348105    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:27:45.362826 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:27:45.349836    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:28:00.790450 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:00.790416    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-688979447f-bb69d"]
Jul 31 14:28:25.810887 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:25.810827    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-688979447f-bb69d" podUID="e29c3784-070c-4687-8206-fab5d3eb778f" containerName="console" containerID="cri-o://907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878" gracePeriod=15
Jul 31 14:28:26.071733 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.071661    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:28:26.187661 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.187627    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-trusted-ca-bundle\") pod \"e29c3784-070c-4687-8206-fab5d3eb778f\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") "
Jul 31 14:28:26.187853 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.187723    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-console-config\") pod \"e29c3784-070c-4687-8206-fab5d3eb778f\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") "
Jul 31 14:28:26.187853 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.187775    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-serving-cert\") pod \"e29c3784-070c-4687-8206-fab5d3eb778f\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") "
Jul 31 14:28:26.187853 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.187793    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-service-ca\") pod \"e29c3784-070c-4687-8206-fab5d3eb778f\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") "
Jul 31 14:28:26.187853 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.187810    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-oauth-serving-cert\") pod \"e29c3784-070c-4687-8206-fab5d3eb778f\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") "
Jul 31 14:28:26.188120 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.187859    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9l7ff\" (UniqueName: \"kubernetes.io/projected/e29c3784-070c-4687-8206-fab5d3eb778f-kube-api-access-9l7ff\") pod \"e29c3784-070c-4687-8206-fab5d3eb778f\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") "
Jul 31 14:28:26.188120 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.187879    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-oauth-config\") pod \"e29c3784-070c-4687-8206-fab5d3eb778f\" (UID: \"e29c3784-070c-4687-8206-fab5d3eb778f\") "
Jul 31 14:28:26.188230 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.188150    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e29c3784-070c-4687-8206-fab5d3eb778f" (UID: "e29c3784-070c-4687-8206-fab5d3eb778f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:28:26.188286 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.188232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-service-ca" (OuterVolumeSpecName: "service-ca") pod "e29c3784-070c-4687-8206-fab5d3eb778f" (UID: "e29c3784-070c-4687-8206-fab5d3eb778f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:28:26.188286 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.188230    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-console-config" (OuterVolumeSpecName: "console-config") pod "e29c3784-070c-4687-8206-fab5d3eb778f" (UID: "e29c3784-070c-4687-8206-fab5d3eb778f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:28:26.188544 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.188520    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e29c3784-070c-4687-8206-fab5d3eb778f" (UID: "e29c3784-070c-4687-8206-fab5d3eb778f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:28:26.189953 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.189932    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e29c3784-070c-4687-8206-fab5d3eb778f" (UID: "e29c3784-070c-4687-8206-fab5d3eb778f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:28:26.190588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.190569    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e29c3784-070c-4687-8206-fab5d3eb778f" (UID: "e29c3784-070c-4687-8206-fab5d3eb778f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:28:26.190588 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.190578    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29c3784-070c-4687-8206-fab5d3eb778f-kube-api-access-9l7ff" (OuterVolumeSpecName: "kube-api-access-9l7ff") pod "e29c3784-070c-4687-8206-fab5d3eb778f" (UID: "e29c3784-070c-4687-8206-fab5d3eb778f"). InnerVolumeSpecName "kube-api-access-9l7ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:28:26.288927 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.288897    2558 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-console-config\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:28:26.288927 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.288919    2558 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-serving-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:28:26.288927 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.288929    2558 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:28:26.289168 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.288940    2558 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-oauth-serving-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:28:26.289168 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.288949    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l7ff\" (UniqueName: \"kubernetes.io/projected/e29c3784-070c-4687-8206-fab5d3eb778f-kube-api-access-9l7ff\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:28:26.289168 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.288957    2558 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e29c3784-070c-4687-8206-fab5d3eb778f-console-oauth-config\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:28:26.289168 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.288966    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e29c3784-070c-4687-8206-fab5d3eb778f-trusted-ca-bundle\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:28:26.613129 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.613042    2558 generic.go:356] "Generic (PLEG): container finished" podID="7c5561a1-775e-4ccb-a096-19bbd398bde0" containerID="7a06779a857445d5e1ce0e5a3f47edb09a490a23c88a80d50af6ea0ae8edda00" exitCode=0
Jul 31 14:28:26.613129 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.613109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" event={"ID":"7c5561a1-775e-4ccb-a096-19bbd398bde0","Type":"ContainerDied","Data":"7a06779a857445d5e1ce0e5a3f47edb09a490a23c88a80d50af6ea0ae8edda00"}
Jul 31 14:28:26.613648 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.613619    2558 scope.go:117] "RemoveContainer" containerID="7a06779a857445d5e1ce0e5a3f47edb09a490a23c88a80d50af6ea0ae8edda00"
Jul 31 14:28:26.614436 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.614416    2558 generic.go:356] "Generic (PLEG): container finished" podID="e29c3784-070c-4687-8206-fab5d3eb778f" containerID="907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878" exitCode=0
Jul 31 14:28:26.614521 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.614463    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-688979447f-bb69d"
Jul 31 14:28:26.614521 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.614502    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688979447f-bb69d" event={"ID":"e29c3784-070c-4687-8206-fab5d3eb778f","Type":"ContainerDied","Data":"907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878"}
Jul 31 14:28:26.614616 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.614540    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-688979447f-bb69d" event={"ID":"e29c3784-070c-4687-8206-fab5d3eb778f","Type":"ContainerDied","Data":"ef6708c0d85dd01366462465eaa8995b3105ae4ac249d93a14fb24d40534ce7e"}
Jul 31 14:28:26.614616 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.614562    2558 scope.go:117] "RemoveContainer" containerID="907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878"
Jul 31 14:28:26.614716 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.614682    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:28:26.614873 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.614771    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:28:26.614873 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.614817    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:28:26.625888 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.625869    2558 scope.go:117] "RemoveContainer" containerID="907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878"
Jul 31 14:28:26.626199 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:28:26.626178    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878\": container with ID starting with 907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878 not found: ID does not exist" containerID="907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878"
Jul 31 14:28:26.626292 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.626205    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878"} err="failed to get container status \"907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878\": rpc error: code = NotFound desc = could not find container \"907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878\": container with ID starting with 907ab46c4cc9da2807a20ca4426b7864a5a7f1ba174a17b09b3cf066a6938878 not found: ID does not exist"
Jul 31 14:28:26.650706 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.650654    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-688979447f-bb69d"]
Jul 31 14:28:26.652787 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:26.652767    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-688979447f-bb69d"]
Jul 31 14:28:27.580126 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:27.580097    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29c3784-070c-4687-8206-fab5d3eb778f" path="/var/lib/kubelet/pods/e29c3784-070c-4687-8206-fab5d3eb778f/volumes"
Jul 31 14:28:27.624238 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:27.622835    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" event={"ID":"7c5561a1-775e-4ccb-a096-19bbd398bde0","Type":"ContainerStarted","Data":"e2dd4fe8e0b9c40ccc5d690c17c100018ceaefadcd87f877deb00e8944b3797d"}
Jul 31 14:28:28.379057 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:28.378998    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:28:28.379290 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:28.379267    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-x79gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 31 14:28:28.379383 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:28.379325    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" podUID="7c5561a1-775e-4ccb-a096-19bbd398bde0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 31 14:28:29.379593 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:29.379557    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-x79gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 31 14:28:29.380003 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:29.379619    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" podUID="7c5561a1-775e-4ccb-a096-19bbd398bde0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 31 14:28:30.378828 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:30.378792    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-x79gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 31 14:28:30.379006 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:30.378850    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" podUID="7c5561a1-775e-4ccb-a096-19bbd398bde0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 31 14:28:31.379327 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:31.379287    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-x79gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 31 14:28:31.379785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:31.379347    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" podUID="7c5561a1-775e-4ccb-a096-19bbd398bde0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 31 14:28:32.379665 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:32.379628    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-x79gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 31 14:28:32.380083 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:32.379692    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg" podUID="7c5561a1-775e-4ccb-a096-19bbd398bde0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 31 14:28:33.382663 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:33.382633    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:28:33.383078 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:33.382893    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:28:33.383687 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:28:33.383662    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-x79gg"
Jul 31 14:29:04.273829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.273746    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"]
Jul 31 14:29:04.274302 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.274239    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29c3784-070c-4687-8206-fab5d3eb778f" containerName="console"
Jul 31 14:29:04.274302 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.274252    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29c3784-070c-4687-8206-fab5d3eb778f" containerName="console"
Jul 31 14:29:04.274377 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.274318    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29c3784-070c-4687-8206-fab5d3eb778f" containerName="console"
Jul 31 14:29:04.277177 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.277160    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.281166 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.281144    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7xdgk\""
Jul 31 14:29:04.281299 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.281165    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 31 14:29:04.281299 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.281178    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 31 14:29:04.281299 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.281215    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 31 14:29:04.284333 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.284309    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"]
Jul 31 14:29:04.347940 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.347900    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.348146 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.347959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjwc5\" (UniqueName: \"kubernetes.io/projected/00856855-cb2f-4ecd-8665-bbd34dd894ab-kube-api-access-gjwc5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.348146 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.348005    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.348146 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.348077    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.348146 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.348100    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00856855-cb2f-4ecd-8665-bbd34dd894ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.348318 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.348229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.448946 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.448917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.449155 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.448960    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.449155 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.448987    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwc5\" (UniqueName: \"kubernetes.io/projected/00856855-cb2f-4ecd-8665-bbd34dd894ab-kube-api-access-gjwc5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.449155 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.449097    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.449155 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.449134    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.449388 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.449158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00856855-cb2f-4ecd-8665-bbd34dd894ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.449447 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.449410    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.449542 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.449515    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.449601 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.449575    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.451389 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.451361    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00856855-cb2f-4ecd-8665-bbd34dd894ab-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.451614 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.451597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00856855-cb2f-4ecd-8665-bbd34dd894ab-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.457834 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.457812    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwc5\" (UniqueName: \"kubernetes.io/projected/00856855-cb2f-4ecd-8665-bbd34dd894ab-kube-api-access-gjwc5\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qspdj\" (UID: \"00856855-cb2f-4ecd-8665-bbd34dd894ab\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.588981 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.588949    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:04.726972 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.726942    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"]
Jul 31 14:29:04.728241 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:29:04.728211    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00856855_cb2f_4ecd_8665_bbd34dd894ab.slice/crio-81a16eaa59a989084070717d1e38ff48ad34952d6fa98b0bd64d82b32484e8a3 WatchSource:0}: Error finding container 81a16eaa59a989084070717d1e38ff48ad34952d6fa98b0bd64d82b32484e8a3: Status 404 returned error can't find the container with id 81a16eaa59a989084070717d1e38ff48ad34952d6fa98b0bd64d82b32484e8a3
Jul 31 14:29:04.729702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.729684    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:29:04.834469 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:04.834431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj" event={"ID":"00856855-cb2f-4ecd-8665-bbd34dd894ab","Type":"ContainerStarted","Data":"81a16eaa59a989084070717d1e38ff48ad34952d6fa98b0bd64d82b32484e8a3"}
Jul 31 14:29:05.841493 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:05.841456    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj" event={"ID":"00856855-cb2f-4ecd-8665-bbd34dd894ab","Type":"ContainerStarted","Data":"a2df68844aeb84c20ce47e99a11fcd1e141feecbe71bbb1badb4223e1f0ee57e"}
Jul 31 14:29:05.863719 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:05.863660    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj" podStartSLOduration=0.893836794 podStartE2EDuration="1.863641808s" totalImagesPullingTime="969.805014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:29:05.861204742 +0000 UTC m=+984.733767996" watchObservedRunningTime="2026-07-31 14:29:05.863641808 +0000 UTC m=+984.736205061"
Jul 31 14:29:14.589228 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:14.589190    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:14.589641 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:14.589245    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:14.601889 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:14.601861    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:14.896539 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:14.896451    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qspdj"
Jul 31 14:29:18.464980 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.464920    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"]
Jul 31 14:29:18.468239 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.468219    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.470680 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.470658    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 31 14:29:18.480437 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.480415    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"]
Jul 31 14:29:18.582635 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.582594    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.582829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.582657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmdtc\" (UniqueName: \"kubernetes.io/projected/223d2c47-0bf4-44d6-86a1-e68c18287892-kube-api-access-mmdtc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.582829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.582699    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.582829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.582729    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223d2c47-0bf4-44d6-86a1-e68c18287892-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.582829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.582761    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.582829 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.582788    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.683979 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.683933    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.684217 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.684001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223d2c47-0bf4-44d6-86a1-e68c18287892-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.684217 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.684089    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.684358 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.684250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.684446 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.684420    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.684517 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.684496    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.684589 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.684573    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmdtc\" (UniqueName: \"kubernetes.io/projected/223d2c47-0bf4-44d6-86a1-e68c18287892-kube-api-access-mmdtc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.684655 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.684588    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.684808 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.684789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.686446 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.686418    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/223d2c47-0bf4-44d6-86a1-e68c18287892-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.686633 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.686613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/223d2c47-0bf4-44d6-86a1-e68c18287892-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.692210 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.692185    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmdtc\" (UniqueName: \"kubernetes.io/projected/223d2c47-0bf4-44d6-86a1-e68c18287892-kube-api-access-mmdtc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dw27j\" (UID: \"223d2c47-0bf4-44d6-86a1-e68c18287892\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.778998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.778901    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:18.914623 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:18.914595    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"]
Jul 31 14:29:18.916175 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:29:18.916145    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223d2c47_0bf4_44d6_86a1_e68c18287892.slice/crio-932c2c2ef7cfe5978f61680cf6449a21e4c848ddc33f8e3cbeebe48d65a7be93 WatchSource:0}: Error finding container 932c2c2ef7cfe5978f61680cf6449a21e4c848ddc33f8e3cbeebe48d65a7be93: Status 404 returned error can't find the container with id 932c2c2ef7cfe5978f61680cf6449a21e4c848ddc33f8e3cbeebe48d65a7be93
Jul 31 14:29:19.913088 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:19.913052    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j" event={"ID":"223d2c47-0bf4-44d6-86a1-e68c18287892","Type":"ContainerStarted","Data":"8210696234f6a62215735daf01855892eeef0b2fbc4515c09c4c5b2ac8d11870"}
Jul 31 14:29:19.913546 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:19.913116    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j" event={"ID":"223d2c47-0bf4-44d6-86a1-e68c18287892","Type":"ContainerStarted","Data":"932c2c2ef7cfe5978f61680cf6449a21e4c848ddc33f8e3cbeebe48d65a7be93"}
Jul 31 14:29:19.933285 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:19.933228    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j" podStartSLOduration=1.7426626440000001 podStartE2EDuration="1.9332116s" totalImagesPullingTime="190.548956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:29:19.930400573 +0000 UTC m=+998.802963846" watchObservedRunningTime="2026-07-31 14:29:19.9332116 +0000 UTC m=+998.805774853"
Jul 31 14:29:28.779133 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:28.779097    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:28.779541 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:28.779157    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:28.791867 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:28.791837    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:29:28.975414 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:29:28.975386    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dw27j"
Jul 31 14:30:00.140503 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:00.140467    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"]
Jul 31 14:30:00.144130 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:00.144113    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"
Jul 31 14:30:00.146640 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:00.146617    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 14:30:00.150923 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:00.150897    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"]
Jul 31 14:30:00.156200 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:00.156176    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"
Jul 31 14:30:00.289954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:00.289931    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"]
Jul 31 14:30:00.292747 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:30:00.292717    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b30599_5e24_40a8_8f6f_548023e803be.slice/crio-9292050abf89ea4d69aab40d04dc76455ced5c3081e8cae8f69a2e23e03cc05e WatchSource:0}: Error finding container 9292050abf89ea4d69aab40d04dc76455ced5c3081e8cae8f69a2e23e03cc05e: Status 404 returned error can't find the container with id 9292050abf89ea4d69aab40d04dc76455ced5c3081e8cae8f69a2e23e03cc05e
Jul 31 14:30:01.137743 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:01.137702    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr" event={"ID":"67b30599-5e24-40a8-8f6f-548023e803be","Type":"ContainerStarted","Data":"9292050abf89ea4d69aab40d04dc76455ced5c3081e8cae8f69a2e23e03cc05e"}
Jul 31 14:30:04.156957 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:04.156920    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr" event={"ID":"67b30599-5e24-40a8-8f6f-548023e803be","Type":"ContainerStarted","Data":"c23f9176909439a9dbb4bea348fd60aa887ccb4bd56ac55274de431b46645d4e"}
Jul 31 14:30:04.173354 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:04.173300    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr" podStartSLOduration=0.721867456 podStartE2EDuration="4.173283726s" totalImagesPullingTime="3.45141627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:30:04.17107589 +0000 UTC m=+1043.043639143" watchObservedRunningTime="2026-07-31 14:30:04.173283726 +0000 UTC m=+1043.045846978"
Jul 31 14:30:05.163930 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:05.163890    2558 generic.go:356] "Generic (PLEG): container finished" podID="67b30599-5e24-40a8-8f6f-548023e803be" containerID="c23f9176909439a9dbb4bea348fd60aa887ccb4bd56ac55274de431b46645d4e" exitCode=0
Jul 31 14:30:05.164351 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:05.163962    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr" event={"ID":"67b30599-5e24-40a8-8f6f-548023e803be","Type":"ContainerDied","Data":"c23f9176909439a9dbb4bea348fd60aa887ccb4bd56ac55274de431b46645d4e"}
Jul 31 14:30:06.305051 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:06.305014    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"
Jul 31 14:30:07.176089 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:07.176058    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"
Jul 31 14:30:07.176276 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:07.176057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr" event={"ID":"67b30599-5e24-40a8-8f6f-548023e803be","Type":"ContainerDied","Data":"9292050abf89ea4d69aab40d04dc76455ced5c3081e8cae8f69a2e23e03cc05e"}
Jul 31 14:30:07.176276 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:07.176185    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9292050abf89ea4d69aab40d04dc76455ced5c3081e8cae8f69a2e23e03cc05e"
Jul 31 14:30:32.588186 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:32.588146    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fb488b8bd-n26kn"]
Jul 31 14:30:32.588711 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:32.588383    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6fb488b8bd-n26kn" podUID="dd09525a-39b8-4cbc-a844-7a5006a33717" containerName="authorino" containerID="cri-o://1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e" gracePeriod=30
Jul 31 14:30:32.842889 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:32.842829    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:30:32.957332 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:32.957299    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd09525a-39b8-4cbc-a844-7a5006a33717-tls-cert\") pod \"dd09525a-39b8-4cbc-a844-7a5006a33717\" (UID: \"dd09525a-39b8-4cbc-a844-7a5006a33717\") "
Jul 31 14:30:32.957517 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:32.957343    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sd69\" (UniqueName: \"kubernetes.io/projected/dd09525a-39b8-4cbc-a844-7a5006a33717-kube-api-access-2sd69\") pod \"dd09525a-39b8-4cbc-a844-7a5006a33717\" (UID: \"dd09525a-39b8-4cbc-a844-7a5006a33717\") "
Jul 31 14:30:32.959415 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:32.959382    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd09525a-39b8-4cbc-a844-7a5006a33717-kube-api-access-2sd69" (OuterVolumeSpecName: "kube-api-access-2sd69") pod "dd09525a-39b8-4cbc-a844-7a5006a33717" (UID: "dd09525a-39b8-4cbc-a844-7a5006a33717"). InnerVolumeSpecName "kube-api-access-2sd69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:30:32.968920 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:32.968895    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd09525a-39b8-4cbc-a844-7a5006a33717-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dd09525a-39b8-4cbc-a844-7a5006a33717" (UID: "dd09525a-39b8-4cbc-a844-7a5006a33717"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:30:33.058077 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.058019    2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dd09525a-39b8-4cbc-a844-7a5006a33717-tls-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:30:33.058077 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.058074    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sd69\" (UniqueName: \"kubernetes.io/projected/dd09525a-39b8-4cbc-a844-7a5006a33717-kube-api-access-2sd69\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:30:33.316473 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.316391    2558 generic.go:356] "Generic (PLEG): container finished" podID="dd09525a-39b8-4cbc-a844-7a5006a33717" containerID="1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e" exitCode=0
Jul 31 14:30:33.316473 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.316435    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fb488b8bd-n26kn"
Jul 31 14:30:33.316732 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.316490    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fb488b8bd-n26kn" event={"ID":"dd09525a-39b8-4cbc-a844-7a5006a33717","Type":"ContainerDied","Data":"1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e"}
Jul 31 14:30:33.316732 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.316541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fb488b8bd-n26kn" event={"ID":"dd09525a-39b8-4cbc-a844-7a5006a33717","Type":"ContainerDied","Data":"ebf84191688f4f2db0aabf0d216d4d971d3b8a95682bb4f9c6032dc569126d52"}
Jul 31 14:30:33.316732 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.316565    2558 scope.go:117] "RemoveContainer" containerID="1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e"
Jul 31 14:30:33.326319 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.326154    2558 scope.go:117] "RemoveContainer" containerID="1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e"
Jul 31 14:30:33.326676 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:30:33.326652    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e\": container with ID starting with 1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e not found: ID does not exist" containerID="1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e"
Jul 31 14:30:33.326786 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.326682    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e"} err="failed to get container status \"1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e\": rpc error: code = NotFound desc = could not find container \"1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e\": container with ID starting with 1d43b1fc622924bc4df8c89a486546d40adc9899a02fd6841b672892e8ef315e not found: ID does not exist"
Jul 31 14:30:33.339567 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.339544    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fb488b8bd-n26kn"]
Jul 31 14:30:33.343137 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.343112    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6fb488b8bd-n26kn"]
Jul 31 14:30:33.580838 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:30:33.580760    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd09525a-39b8-4cbc-a844-7a5006a33717" path="/var/lib/kubelet/pods/dd09525a-39b8-4cbc-a844-7a5006a33717/volumes"
Jul 31 14:32:45.380086 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:32:45.380059    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:32:45.382633 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:32:45.382612    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:37:45.411479 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:37:45.411444    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:37:45.415666 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:37:45.415636    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:39:18.554071 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.554040    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"]
Jul 31 14:39:18.554583 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.554243    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8" podUID="e7672885-b77f-4580-ad9c-2dfcfe1e94e6" containerName="manager" containerID="cri-o://48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f" gracePeriod=10
Jul 31 14:39:18.818581 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.818521    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:39:18.894451 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.894421    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9pgh\" (UniqueName: \"kubernetes.io/projected/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-kube-api-access-l9pgh\") pod \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\" (UID: \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\") "
Jul 31 14:39:18.894626 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.894469    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-extensions-socket-volume\") pod \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\" (UID: \"e7672885-b77f-4580-ad9c-2dfcfe1e94e6\") "
Jul 31 14:39:18.894901 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.894878    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e7672885-b77f-4580-ad9c-2dfcfe1e94e6" (UID: "e7672885-b77f-4580-ad9c-2dfcfe1e94e6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:39:18.896537 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.896503    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-kube-api-access-l9pgh" (OuterVolumeSpecName: "kube-api-access-l9pgh") pod "e7672885-b77f-4580-ad9c-2dfcfe1e94e6" (UID: "e7672885-b77f-4580-ad9c-2dfcfe1e94e6"). InnerVolumeSpecName "kube-api-access-l9pgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:39:18.995279 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.995251    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-extensions-socket-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:39:18.995279 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:18.995276    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9pgh\" (UniqueName: \"kubernetes.io/projected/e7672885-b77f-4580-ad9c-2dfcfe1e94e6-kube-api-access-l9pgh\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:39:19.105367 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.105284    2558 generic.go:356] "Generic (PLEG): container finished" podID="e7672885-b77f-4580-ad9c-2dfcfe1e94e6" containerID="48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f" exitCode=0
Jul 31 14:39:19.105367 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.105336    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"
Jul 31 14:39:19.105531 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.105370    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8" event={"ID":"e7672885-b77f-4580-ad9c-2dfcfe1e94e6","Type":"ContainerDied","Data":"48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f"}
Jul 31 14:39:19.105531 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.105427    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8" event={"ID":"e7672885-b77f-4580-ad9c-2dfcfe1e94e6","Type":"ContainerDied","Data":"4bf369f952136a98373b53736edbf65fa2edb531e297a5018f5108e726e98ceb"}
Jul 31 14:39:19.105531 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.105450    2558 scope.go:117] "RemoveContainer" containerID="48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f"
Jul 31 14:39:19.115216 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.115197    2558 scope.go:117] "RemoveContainer" containerID="48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f"
Jul 31 14:39:19.115484 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:39:19.115465    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f\": container with ID starting with 48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f not found: ID does not exist" containerID="48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f"
Jul 31 14:39:19.115534 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.115491    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f"} err="failed to get container status \"48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f\": rpc error: code = NotFound desc = could not find container \"48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f\": container with ID starting with 48ea9f99cff2d0b4f7a9af41b3dc3b01a1669d0f01ecfc127ddab27181c40d5f not found: ID does not exist"
Jul 31 14:39:19.127581 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.127557    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"]
Jul 31 14:39:19.132450 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.132431    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ltfr8"]
Jul 31 14:39:19.580086 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:39:19.580055    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7672885-b77f-4580-ad9c-2dfcfe1e94e6" path="/var/lib/kubelet/pods/e7672885-b77f-4580-ad9c-2dfcfe1e94e6/volumes"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.271122    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"]
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272179    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd09525a-39b8-4cbc-a844-7a5006a33717" containerName="authorino"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272200    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd09525a-39b8-4cbc-a844-7a5006a33717" containerName="authorino"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272242    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7672885-b77f-4580-ad9c-2dfcfe1e94e6" containerName="manager"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272251    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7672885-b77f-4580-ad9c-2dfcfe1e94e6" containerName="manager"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272264    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b30599-5e24-40a8-8f6f-548023e803be" containerName="cleanup"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272273    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b30599-5e24-40a8-8f6f-548023e803be" containerName="cleanup"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272454    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd09525a-39b8-4cbc-a844-7a5006a33717" containerName="authorino"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272477    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7672885-b77f-4580-ad9c-2dfcfe1e94e6" containerName="manager"
Jul 31 14:40:19.275070 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.272492    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b30599-5e24-40a8-8f6f-548023e803be" containerName="cleanup"
Jul 31 14:40:19.276602 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.276573    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:19.279667 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.279637    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q27fh\""
Jul 31 14:40:19.287210 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.287187    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"]
Jul 31 14:40:19.342882 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.342850    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6m9\" (UniqueName: \"kubernetes.io/projected/82771e8a-ad35-41df-b961-ef816c1fff80-kube-api-access-cf6m9\") pod \"kuadrant-operator-controller-manager-68cf589589-2mlrs\" (UID: \"82771e8a-ad35-41df-b961-ef816c1fff80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:19.343064 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.342895    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82771e8a-ad35-41df-b961-ef816c1fff80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-2mlrs\" (UID: \"82771e8a-ad35-41df-b961-ef816c1fff80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:19.443797 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.443765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82771e8a-ad35-41df-b961-ef816c1fff80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-2mlrs\" (UID: \"82771e8a-ad35-41df-b961-ef816c1fff80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:19.443971 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.443879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6m9\" (UniqueName: \"kubernetes.io/projected/82771e8a-ad35-41df-b961-ef816c1fff80-kube-api-access-cf6m9\") pod \"kuadrant-operator-controller-manager-68cf589589-2mlrs\" (UID: \"82771e8a-ad35-41df-b961-ef816c1fff80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:19.444181 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.444160    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82771e8a-ad35-41df-b961-ef816c1fff80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-2mlrs\" (UID: \"82771e8a-ad35-41df-b961-ef816c1fff80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:19.451996 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.451974    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6m9\" (UniqueName: \"kubernetes.io/projected/82771e8a-ad35-41df-b961-ef816c1fff80-kube-api-access-cf6m9\") pod \"kuadrant-operator-controller-manager-68cf589589-2mlrs\" (UID: \"82771e8a-ad35-41df-b961-ef816c1fff80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:19.588572 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.588547    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:19.727942 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:19.727917    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"]
Jul 31 14:40:19.730148 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:40:19.730122    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82771e8a_ad35_41df_b961_ef816c1fff80.slice/crio-66f7ff4a9ee871104fa18181f8de43692e9865aaa0e49fa8e55e061b8611193f WatchSource:0}: Error finding container 66f7ff4a9ee871104fa18181f8de43692e9865aaa0e49fa8e55e061b8611193f: Status 404 returned error can't find the container with id 66f7ff4a9ee871104fa18181f8de43692e9865aaa0e49fa8e55e061b8611193f
Jul 31 14:40:20.429488 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:20.429455    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs" event={"ID":"82771e8a-ad35-41df-b961-ef816c1fff80","Type":"ContainerStarted","Data":"8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100"}
Jul 31 14:40:20.429960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:20.429512    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs" event={"ID":"82771e8a-ad35-41df-b961-ef816c1fff80","Type":"ContainerStarted","Data":"66f7ff4a9ee871104fa18181f8de43692e9865aaa0e49fa8e55e061b8611193f"}
Jul 31 14:40:20.429960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:20.429653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:40:20.450465 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:20.450419    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs" podStartSLOduration=1.450404537 podStartE2EDuration="1.450404537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:40:20.447787994 +0000 UTC m=+1659.320351248" watchObservedRunningTime="2026-07-31 14:40:20.450404537 +0000 UTC m=+1659.322967789"
Jul 31 14:40:31.437904 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:40:31.437867    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:42:45.443785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:42:45.443754    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:42:45.449918 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:42:45.449893    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:44:14.586885 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.586852    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"]
Jul 31 14:44:14.590323 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.590300    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.593061 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.593038    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6b2a4720-kserve-self-signed-certs\""
Jul 31 14:44:14.603559 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.603534    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"]
Jul 31 14:44:14.632285 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.632259    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace17ab2-2738-4caa-8303-c6f3699006b3-tls-certs\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.632456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.632312    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-home\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.632456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.632336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-dshm\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.632456 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.632436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-tmp-dir\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.632635 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.632491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmjh\" (UniqueName: \"kubernetes.io/projected/ace17ab2-2738-4caa-8303-c6f3699006b3-kube-api-access-rhmjh\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.632635 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.632554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-model-cache\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.733415 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.733382    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-home\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.733415 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.733415    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-dshm\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.733694 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.733459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-tmp-dir\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.733694 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.733492    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmjh\" (UniqueName: \"kubernetes.io/projected/ace17ab2-2738-4caa-8303-c6f3699006b3-kube-api-access-rhmjh\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.733694 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.733523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-model-cache\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.733694 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.733564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace17ab2-2738-4caa-8303-c6f3699006b3-tls-certs\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.733954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.733929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-model-cache\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.734014 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.733982    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-home\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.734014 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.734003    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-tmp-dir\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.735703 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.735681    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-dshm\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.735975 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.735959    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace17ab2-2738-4caa-8303-c6f3699006b3-tls-certs\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.741376 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.741352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmjh\" (UniqueName: \"kubernetes.io/projected/ace17ab2-2738-4caa-8303-c6f3699006b3-kube-api-access-rhmjh\") pod \"e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") " pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:14.901323 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:14.901244    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:15.245799 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:15.244419    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"]
Jul 31 14:44:15.246336 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:44:15.246303    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podace17ab2_2738_4caa_8303_c6f3699006b3.slice/crio-e963e11f53b2308fc6a89214aa5384afe959e14bb821e6c3c4d45eadc1abff86 WatchSource:0}: Error finding container e963e11f53b2308fc6a89214aa5384afe959e14bb821e6c3c4d45eadc1abff86: Status 404 returned error can't find the container with id e963e11f53b2308fc6a89214aa5384afe959e14bb821e6c3c4d45eadc1abff86
Jul 31 14:44:15.651194 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:15.651161    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v" event={"ID":"ace17ab2-2738-4caa-8303-c6f3699006b3","Type":"ContainerStarted","Data":"1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d"}
Jul 31 14:44:15.651194 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:15.651205    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v" event={"ID":"ace17ab2-2738-4caa-8303-c6f3699006b3","Type":"ContainerStarted","Data":"e963e11f53b2308fc6a89214aa5384afe959e14bb821e6c3c4d45eadc1abff86"}
Jul 31 14:44:15.669133 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:15.669085    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v" podStartSLOduration=1.6690679739999998 podStartE2EDuration="1.669067974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:44:15.667049495 +0000 UTC m=+1894.539612748" watchObservedRunningTime="2026-07-31 14:44:15.669067974 +0000 UTC m=+1894.541631227"
Jul 31 14:44:24.901890 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:24.901798    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:24.901890 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:24.901840    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:24.914330 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:24.914311    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:44:25.715776 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:44:25.715749    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:45:00.133880 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.133841    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52"]
Jul 31 14:45:00.137437 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.137420    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52"
Jul 31 14:45:00.139998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.139980    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 14:45:00.144071 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.144046    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52"]
Jul 31 14:45:00.148429 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.148407    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52"
Jul 31 14:45:00.490686 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.490663    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52"]
Jul 31 14:45:00.492519 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:45:00.492492    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f97972_a6e1_4b37_8a44_35c1cc929cef.slice/crio-c8e5f762780385e3691ace43471a7531ad97ffc27a998d4680e16539274f61a0 WatchSource:0}: Error finding container c8e5f762780385e3691ace43471a7531ad97ffc27a998d4680e16539274f61a0: Status 404 returned error can't find the container with id c8e5f762780385e3691ace43471a7531ad97ffc27a998d4680e16539274f61a0
Jul 31 14:45:00.891913 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.891868    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52" event={"ID":"d2f97972-a6e1-4b37-8a44-35c1cc929cef","Type":"ContainerStarted","Data":"fa94cb40aeb2b3b0c288297233b054b64f2fd1e3d92521d4e0f45c58ae51e4e7"}
Jul 31 14:45:00.892115 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.891939    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52" event={"ID":"d2f97972-a6e1-4b37-8a44-35c1cc929cef","Type":"ContainerStarted","Data":"c8e5f762780385e3691ace43471a7531ad97ffc27a998d4680e16539274f61a0"}
Jul 31 14:45:00.907505 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:00.907454    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52" podStartSLOduration=0.907437204 podStartE2EDuration="907.437204ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:45:00.906065826 +0000 UTC m=+1939.778629098" watchObservedRunningTime="2026-07-31 14:45:00.907437204 +0000 UTC m=+1939.780000457"
Jul 31 14:45:01.898234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:01.898200    2558 generic.go:356] "Generic (PLEG): container finished" podID="d2f97972-a6e1-4b37-8a44-35c1cc929cef" containerID="fa94cb40aeb2b3b0c288297233b054b64f2fd1e3d92521d4e0f45c58ae51e4e7" exitCode=0
Jul 31 14:45:01.898633 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:01.898295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52" event={"ID":"d2f97972-a6e1-4b37-8a44-35c1cc929cef","Type":"ContainerDied","Data":"fa94cb40aeb2b3b0c288297233b054b64f2fd1e3d92521d4e0f45c58ae51e4e7"}
Jul 31 14:45:03.045813 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:03.045786    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52"
Jul 31 14:45:03.911608 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:03.911523    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52"
Jul 31 14:45:03.911608 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:03.911551    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758485-ddw52" event={"ID":"d2f97972-a6e1-4b37-8a44-35c1cc929cef","Type":"ContainerDied","Data":"c8e5f762780385e3691ace43471a7531ad97ffc27a998d4680e16539274f61a0"}
Jul 31 14:45:03.911608 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:03.911604    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8e5f762780385e3691ace43471a7531ad97ffc27a998d4680e16539274f61a0"
Jul 31 14:45:14.979755 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:14.979715    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"]
Jul 31 14:45:29.999398 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:29.999358    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v" podUID="ace17ab2-2738-4caa-8303-c6f3699006b3" containerName="main" containerID="cri-o://1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d" gracePeriod=45
Jul 31 14:45:30.255978 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.255912    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:45:30.349597 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349563    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-home\") pod \"ace17ab2-2738-4caa-8303-c6f3699006b3\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") "
Jul 31 14:45:30.349597 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349599    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhmjh\" (UniqueName: \"kubernetes.io/projected/ace17ab2-2738-4caa-8303-c6f3699006b3-kube-api-access-rhmjh\") pod \"ace17ab2-2738-4caa-8303-c6f3699006b3\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") "
Jul 31 14:45:30.349848 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349650    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-tmp-dir\") pod \"ace17ab2-2738-4caa-8303-c6f3699006b3\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") "
Jul 31 14:45:30.349848 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349689    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace17ab2-2738-4caa-8303-c6f3699006b3-tls-certs\") pod \"ace17ab2-2738-4caa-8303-c6f3699006b3\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") "
Jul 31 14:45:30.349848 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349714    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-model-cache\") pod \"ace17ab2-2738-4caa-8303-c6f3699006b3\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") "
Jul 31 14:45:30.349848 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349731    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-dshm\") pod \"ace17ab2-2738-4caa-8303-c6f3699006b3\" (UID: \"ace17ab2-2738-4caa-8303-c6f3699006b3\") "
Jul 31 14:45:30.350086 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349904    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-home" (OuterVolumeSpecName: "home") pod "ace17ab2-2738-4caa-8303-c6f3699006b3" (UID: "ace17ab2-2738-4caa-8303-c6f3699006b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:45:30.350086 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349950    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ace17ab2-2738-4caa-8303-c6f3699006b3" (UID: "ace17ab2-2738-4caa-8303-c6f3699006b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:45:30.350086 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349988    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "ace17ab2-2738-4caa-8303-c6f3699006b3" (UID: "ace17ab2-2738-4caa-8303-c6f3699006b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:45:30.350086 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.349998    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:45:30.352013 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.351977    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-dshm" (OuterVolumeSpecName: "dshm") pod "ace17ab2-2738-4caa-8303-c6f3699006b3" (UID: "ace17ab2-2738-4caa-8303-c6f3699006b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:45:30.352013 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.351991    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace17ab2-2738-4caa-8303-c6f3699006b3-kube-api-access-rhmjh" (OuterVolumeSpecName: "kube-api-access-rhmjh") pod "ace17ab2-2738-4caa-8303-c6f3699006b3" (UID: "ace17ab2-2738-4caa-8303-c6f3699006b3"). InnerVolumeSpecName "kube-api-access-rhmjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:45:30.352013 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.351993    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace17ab2-2738-4caa-8303-c6f3699006b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ace17ab2-2738-4caa-8303-c6f3699006b3" (UID: "ace17ab2-2738-4caa-8303-c6f3699006b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:45:30.451351 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.451320    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmjh\" (UniqueName: \"kubernetes.io/projected/ace17ab2-2738-4caa-8303-c6f3699006b3-kube-api-access-rhmjh\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:45:30.451351 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.451346    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:45:30.451351 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.451358    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ace17ab2-2738-4caa-8303-c6f3699006b3-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:45:30.451584 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.451366    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:45:30.451584 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:30.451375    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ace17ab2-2738-4caa-8303-c6f3699006b3-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:45:31.069504 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.069463    2558 generic.go:356] "Generic (PLEG): container finished" podID="ace17ab2-2738-4caa-8303-c6f3699006b3" containerID="1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d" exitCode=0
Jul 31 14:45:31.069954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.069515    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v" event={"ID":"ace17ab2-2738-4caa-8303-c6f3699006b3","Type":"ContainerDied","Data":"1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d"}
Jul 31 14:45:31.069954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.069527    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"
Jul 31 14:45:31.069954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.069556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v" event={"ID":"ace17ab2-2738-4caa-8303-c6f3699006b3","Type":"ContainerDied","Data":"e963e11f53b2308fc6a89214aa5384afe959e14bb821e6c3c4d45eadc1abff86"}
Jul 31 14:45:31.069954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.069578    2558 scope.go:117] "RemoveContainer" containerID="1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d"
Jul 31 14:45:31.079776 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.079759    2558 scope.go:117] "RemoveContainer" containerID="1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d"
Jul 31 14:45:31.080050 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:45:31.080011    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d\": container with ID starting with 1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d not found: ID does not exist" containerID="1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d"
Jul 31 14:45:31.080123 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.080058    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d"} err="failed to get container status \"1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d\": rpc error: code = NotFound desc = could not find container \"1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d\": container with ID starting with 1c00c6c56dc96a28cce9f187e1ab4c1e139cb501e61ecf42f853dab9fb4e714d not found: ID does not exist"
Jul 31 14:45:31.091926 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.091901    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"]
Jul 31 14:45:31.094213 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.094189    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6b2a4720-kserve-768ff5d45f-pnv9v"]
Jul 31 14:45:31.580427 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:45:31.580395    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ace17ab2-2738-4caa-8303-c6f3699006b3" path="/var/lib/kubelet/pods/ace17ab2-2738-4caa-8303-c6f3699006b3/volumes"
Jul 31 14:47:45.476079 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:47:45.476046    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:47:45.483182 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:47:45.483157    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:49:10.900304 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.900223    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"]
Jul 31 14:49:10.900752 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.900640    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f97972-a6e1-4b37-8a44-35c1cc929cef" containerName="cleanup"
Jul 31 14:49:10.900752 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.900652    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f97972-a6e1-4b37-8a44-35c1cc929cef" containerName="cleanup"
Jul 31 14:49:10.900752 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.900672    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ace17ab2-2738-4caa-8303-c6f3699006b3" containerName="main"
Jul 31 14:49:10.900752 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.900678    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ace17ab2-2738-4caa-8303-c6f3699006b3" containerName="main"
Jul 31 14:49:10.900752 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.900749    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f97972-a6e1-4b37-8a44-35c1cc929cef" containerName="cleanup"
Jul 31 14:49:10.900917 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.900760    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ace17ab2-2738-4caa-8303-c6f3699006b3" containerName="main"
Jul 31 14:49:10.903957 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.903942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:10.906754 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.906736    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3ea37d42-openshift-default-dockercfg-4svlv\""
Jul 31 14:49:10.919939 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:10.919902    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"]
Jul 31 14:49:11.021361 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.021304    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-token\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.021361 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.021357    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istiod-ca-cert\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.021624 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.021420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-envoy\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.021624 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.021536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cqcr\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-kube-api-access-9cqcr\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.021624 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.021609    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-credential-socket\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.021785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.021699    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-certs\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.021785 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.021734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-podinfo\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.022255 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.022229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-data\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.022370 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.022285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-socket\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.123702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.123659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cqcr\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-kube-api-access-9cqcr\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.123894 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.123735    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-credential-socket\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.123974 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.123950    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-certs\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124072 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.123973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-podinfo\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124072 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.123996    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-data\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124072 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.124017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-socket\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124243 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.124223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-token\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124300 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.124275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istiod-ca-cert\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124356 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.124324    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-envoy\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124466 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.124443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-certs\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124536 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.124486    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-socket\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124593 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.124534    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-data\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.124830 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.124807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-credential-socket\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.125164 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.125139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istiod-ca-cert\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.126674 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.126655    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-envoy\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.126908 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.126885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-podinfo\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.131877 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.131851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cqcr\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-kube-api-access-9cqcr\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.132220 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.132192    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-token\") pod \"e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") " pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.215537 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.215450    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:11.363538 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.363512    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"]
Jul 31 14:49:11.365189 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:49:11.365158    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19dc1ea2_b8ab_4cd4_adb3_e52756382f3c.slice/crio-639515aa4a25bb1131fb109a030be347955d987c01d41a323b86cb60f097718e WatchSource:0}: Error finding container 639515aa4a25bb1131fb109a030be347955d987c01d41a323b86cb60f097718e: Status 404 returned error can't find the container with id 639515aa4a25bb1131fb109a030be347955d987c01d41a323b86cb60f097718e
Jul 31 14:49:11.367728 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.367683    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:49:11.367843 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.367770    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:49:11.367843 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:11.367811    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:49:12.260420 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:12.260384    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" event={"ID":"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c","Type":"ContainerStarted","Data":"55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86"}
Jul 31 14:49:12.260816 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:12.260465    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" event={"ID":"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c","Type":"ContainerStarted","Data":"639515aa4a25bb1131fb109a030be347955d987c01d41a323b86cb60f097718e"}
Jul 31 14:49:12.282414 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:12.282361    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" podStartSLOduration=2.282347435 podStartE2EDuration="2.282347435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:12.281001172 +0000 UTC m=+2191.153564425" watchObservedRunningTime="2026-07-31 14:49:12.282347435 +0000 UTC m=+2191.154910689"
Jul 31 14:49:13.216168 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:13.216131    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:13.221368 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:13.221336    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:13.266007 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:13.265978    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:13.267048 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:13.267002    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:26.336632 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:26.336606    2558 generic.go:356] "Generic (PLEG): container finished" podID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerID="55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86" exitCode=0
Jul 31 14:49:26.336995 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:26.336647    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" event={"ID":"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c","Type":"ContainerDied","Data":"55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86"}
Jul 31 14:49:26.337116 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:26.337019    2558 scope.go:117] "RemoveContainer" containerID="55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86"
Jul 31 14:49:26.338161 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:26.338132    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:49:26.338226 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:26.338203    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:49:26.338263 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:26.338233    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 14:49:27.352523 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:27.352489    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" event={"ID":"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c","Type":"ContainerStarted","Data":"7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c"}
Jul 31 14:49:28.216615 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:28.216578    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:28.217885 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:28.217855    2558 patch_prober.go:28] interesting pod/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 31 14:49:28.218044 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:28.217916    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 31 14:49:28.266207 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:28.266167    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:29.216497 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:29.216466    2558 patch_prober.go:28] interesting pod/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 31 14:49:29.216957 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:29.216931    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 31 14:49:30.216602 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:30.216566    2558 patch_prober.go:28] interesting pod/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 31 14:49:30.216995 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:30.216625    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 31 14:49:31.216707 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:31.216669    2558 patch_prober.go:28] interesting pod/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 31 14:49:31.217113 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:31.216723    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 31 14:49:32.216402 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:32.216364    2558 patch_prober.go:28] interesting pod/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 31 14:49:32.216594 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:32.216427    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 31 14:49:33.219796 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:33.219764    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:33.220656 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:33.220634    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:49:55.480877 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:55.480840    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"]
Jul 31 14:49:55.481507 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:49:55.481130    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy" containerID="cri-o://7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c" gracePeriod=30
Jul 31 14:50:00.733061 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.733014    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:50:00.828294 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828265    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-envoy\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.828461 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828327    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-token\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.828461 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828372    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-credential-socket\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.828461 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828403    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cqcr\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-kube-api-access-9cqcr\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.828584 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828550    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-certs\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.828638 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-podinfo\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.828700 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828662    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istiod-ca-cert\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.828700 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828688    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-socket\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.828809 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828701    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:50:00.828965 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828925    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-data" (OuterVolumeSpecName: "istio-data") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:50:00.829141 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.829076    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:50:00.829236 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.829094    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:00.829317 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.828714    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-data\") pod \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\" (UID: \"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c\") "
Jul 31 14:50:00.829446 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.829284    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:50:00.829700 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.829681    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:00.829761 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.829707    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:00.829761 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.829722    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:00.829761 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.829735    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:00.829761 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.829749    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:00.830745 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.830719    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 14:50:00.830841 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.830720    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-kube-api-access-9cqcr" (OuterVolumeSpecName: "kube-api-access-9cqcr") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "kube-api-access-9cqcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:00.830956 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.830936    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:50:00.831015 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.830996    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-token" (OuterVolumeSpecName: "istio-token") pod "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" (UID: "19dc1ea2-b8ab-4cd4-adb3-e52756382f3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:00.930360 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.930322    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:00.930360 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.930358    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:00.930564 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.930375    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cqcr\" (UniqueName: \"kubernetes.io/projected/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-kube-api-access-9cqcr\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:00.930564 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:00.930391    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:01.538761 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.538724    2558 generic.go:356] "Generic (PLEG): container finished" podID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerID="7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c" exitCode=0
Jul 31 14:50:01.538950 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.538780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" event={"ID":"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c","Type":"ContainerDied","Data":"7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c"}
Jul 31 14:50:01.538950 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.538797    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"
Jul 31 14:50:01.538950 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.538866    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q" event={"ID":"19dc1ea2-b8ab-4cd4-adb3-e52756382f3c","Type":"ContainerDied","Data":"639515aa4a25bb1131fb109a030be347955d987c01d41a323b86cb60f097718e"}
Jul 31 14:50:01.538950 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.538897    2558 scope.go:117] "RemoveContainer" containerID="7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c"
Jul 31 14:50:01.548562 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.548542    2558 scope.go:117] "RemoveContainer" containerID="55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86"
Jul 31 14:50:01.557220 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.557197    2558 scope.go:117] "RemoveContainer" containerID="7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c"
Jul 31 14:50:01.557521 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:50:01.557485    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c\": container with ID starting with 7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c not found: ID does not exist" containerID="7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c"
Jul 31 14:50:01.557627 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.557533    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c"} err="failed to get container status \"7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c\": rpc error: code = NotFound desc = could not find container \"7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c\": container with ID starting with 7c32383a6bc2e4498cdf617bb6eb5e3d0916a922817515b597932a237c7fb81c not found: ID does not exist"
Jul 31 14:50:01.557627 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.557563    2558 scope.go:117] "RemoveContainer" containerID="55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86"
Jul 31 14:50:01.557902 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:50:01.557874    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86\": container with ID starting with 55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86 not found: ID does not exist" containerID="55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86"
Jul 31 14:50:01.557998 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.557914    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86"} err="failed to get container status \"55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86\": rpc error: code = NotFound desc = could not find container \"55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86\": container with ID starting with 55fdd18eaecdeb4a072da20dbf13737c75ef30c7cf2d355eab0b6947a7c5bb86 not found: ID does not exist"
Jul 31 14:50:01.560570 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.560551    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"]
Jul 31 14:50:01.565629 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.565609    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ea37d42-openshift-default-845c8d55c4-g287q"]
Jul 31 14:50:01.579915 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:01.579893    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" path="/var/lib/kubelet/pods/19dc1ea2-b8ab-4cd4-adb3-e52756382f3c/volumes"
Jul 31 14:50:27.757324 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.757289    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"]
Jul 31 14:50:27.757731 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.757697    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy"
Jul 31 14:50:27.757731 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.757709    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy"
Jul 31 14:50:27.757731 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.757723    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy"
Jul 31 14:50:27.757731 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.757729    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy"
Jul 31 14:50:27.757866 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.757805    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy"
Jul 31 14:50:27.757866 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.757811    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="19dc1ea2-b8ab-4cd4-adb3-e52756382f3c" containerName="istio-proxy"
Jul 31 14:50:27.760861 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.760844    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"
Jul 31 14:50:27.763446 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.763428    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 14:50:27.768307 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.768256    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"]
Jul 31 14:50:27.874424 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.874391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3908fe-0620-4c43-b028-a0b277aa98b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm\" (UID: \"1c3908fe-0620-4c43-b028-a0b277aa98b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"
Jul 31 14:50:27.975103 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.975067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3908fe-0620-4c43-b028-a0b277aa98b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm\" (UID: \"1c3908fe-0620-4c43-b028-a0b277aa98b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"
Jul 31 14:50:27.975752 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:27.975731    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3908fe-0620-4c43-b028-a0b277aa98b6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm\" (UID: \"1c3908fe-0620-4c43-b028-a0b277aa98b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"
Jul 31 14:50:28.073312 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:28.073276    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"
Jul 31 14:50:28.210859 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:28.210829    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"]
Jul 31 14:50:28.212514 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:50:28.212483    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3908fe_0620_4c43_b028_a0b277aa98b6.slice/crio-06c4c11a4a7b7cba0cd14e23773f42c8ad0c8dec0d99a65c910adc86002677cf WatchSource:0}: Error finding container 06c4c11a4a7b7cba0cd14e23773f42c8ad0c8dec0d99a65c910adc86002677cf: Status 404 returned error can't find the container with id 06c4c11a4a7b7cba0cd14e23773f42c8ad0c8dec0d99a65c910adc86002677cf
Jul 31 14:50:28.677263 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:28.677165    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm" event={"ID":"1c3908fe-0620-4c43-b028-a0b277aa98b6","Type":"ContainerStarted","Data":"e374f8605042dbc7ae30c5c333787c94d06925bae26ae5fe56098aa8248c241b"}
Jul 31 14:50:28.677263 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:28.677228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm" event={"ID":"1c3908fe-0620-4c43-b028-a0b277aa98b6","Type":"ContainerStarted","Data":"06c4c11a4a7b7cba0cd14e23773f42c8ad0c8dec0d99a65c910adc86002677cf"}
Jul 31 14:50:28.692923 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:28.692874    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm" podStartSLOduration=1.6928572389999998 podStartE2EDuration="1.692857239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:28.690867783 +0000 UTC m=+2267.563431036" watchObservedRunningTime="2026-07-31 14:50:28.692857239 +0000 UTC m=+2267.565420491"
Jul 31 14:50:29.683531 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:29.683496    2558 generic.go:356] "Generic (PLEG): container finished" podID="1c3908fe-0620-4c43-b028-a0b277aa98b6" containerID="e374f8605042dbc7ae30c5c333787c94d06925bae26ae5fe56098aa8248c241b" exitCode=0
Jul 31 14:50:29.683960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:29.683583    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm" event={"ID":"1c3908fe-0620-4c43-b028-a0b277aa98b6","Type":"ContainerDied","Data":"e374f8605042dbc7ae30c5c333787c94d06925bae26ae5fe56098aa8248c241b"}
Jul 31 14:50:30.824362 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:30.824341    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"
Jul 31 14:50:30.902933 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:30.902899    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3908fe-0620-4c43-b028-a0b277aa98b6-maas-api-service-ca\") pod \"1c3908fe-0620-4c43-b028-a0b277aa98b6\" (UID: \"1c3908fe-0620-4c43-b028-a0b277aa98b6\") "
Jul 31 14:50:30.903326 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:30.903298    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3908fe-0620-4c43-b028-a0b277aa98b6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c3908fe-0620-4c43-b028-a0b277aa98b6" (UID: "1c3908fe-0620-4c43-b028-a0b277aa98b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:31.004431 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:31.004350    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c3908fe-0620-4c43-b028-a0b277aa98b6-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:50:31.695538 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:31.695508    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"
Jul 31 14:50:31.695749 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:31.695555    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm" event={"ID":"1c3908fe-0620-4c43-b028-a0b277aa98b6","Type":"ContainerDied","Data":"06c4c11a4a7b7cba0cd14e23773f42c8ad0c8dec0d99a65c910adc86002677cf"}
Jul 31 14:50:31.695749 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:31.695647    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06c4c11a4a7b7cba0cd14e23773f42c8ad0c8dec0d99a65c910adc86002677cf"
Jul 31 14:50:34.301851 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:34.301813    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"]
Jul 31 14:50:34.306162 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:34.306138    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6abebbcb-b2qbm"]
Jul 31 14:50:35.580156 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:50:35.580124    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c3908fe-0620-4c43-b028-a0b277aa98b6" path="/var/lib/kubelet/pods/1c3908fe-0620-4c43-b028-a0b277aa98b6/volumes"
Jul 31 14:51:38.815289 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:38.815201    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"]
Jul 31 14:51:38.815888 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:38.815868    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c3908fe-0620-4c43-b028-a0b277aa98b6" containerName="revoke-keys"
Jul 31 14:51:38.815980 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:38.815891    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c3908fe-0620-4c43-b028-a0b277aa98b6" containerName="revoke-keys"
Jul 31 14:51:38.816056 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:38.815994    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c3908fe-0620-4c43-b028-a0b277aa98b6" containerName="revoke-keys"
Jul 31 14:51:38.819109 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:38.819088    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"
Jul 31 14:51:38.821632 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:38.821613    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 14:51:38.826239 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:38.826216    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"]
Jul 31 14:51:38.943459 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:38.943423    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23298d5d-9609-4ccd-91b4-d0957aa5b92f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3187acc6-pmx65\" (UID: \"23298d5d-9609-4ccd-91b4-d0957aa5b92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"
Jul 31 14:51:39.044208 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:39.044174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23298d5d-9609-4ccd-91b4-d0957aa5b92f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3187acc6-pmx65\" (UID: \"23298d5d-9609-4ccd-91b4-d0957aa5b92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"
Jul 31 14:51:39.044809 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:39.044790    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23298d5d-9609-4ccd-91b4-d0957aa5b92f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3187acc6-pmx65\" (UID: \"23298d5d-9609-4ccd-91b4-d0957aa5b92f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"
Jul 31 14:51:39.130705 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:39.130626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"
Jul 31 14:51:39.266602 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:39.266577    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"]
Jul 31 14:51:39.268964 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:51:39.268934    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23298d5d_9609_4ccd_91b4_d0957aa5b92f.slice/crio-dc103be63f4e51d4f249faa9ad84486845b06bccae4f96c7661bd979192af702 WatchSource:0}: Error finding container dc103be63f4e51d4f249faa9ad84486845b06bccae4f96c7661bd979192af702: Status 404 returned error can't find the container with id dc103be63f4e51d4f249faa9ad84486845b06bccae4f96c7661bd979192af702
Jul 31 14:51:40.068284 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:40.068248    2558 generic.go:356] "Generic (PLEG): container finished" podID="23298d5d-9609-4ccd-91b4-d0957aa5b92f" containerID="e20ab8f29400997264bc75d39a89c2a904c6290a7d55f43da77abd7694d5c954" exitCode=0
Jul 31 14:51:40.068661 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:40.068300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65" event={"ID":"23298d5d-9609-4ccd-91b4-d0957aa5b92f","Type":"ContainerDied","Data":"e20ab8f29400997264bc75d39a89c2a904c6290a7d55f43da77abd7694d5c954"}
Jul 31 14:51:40.068661 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:40.068351    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65" event={"ID":"23298d5d-9609-4ccd-91b4-d0957aa5b92f","Type":"ContainerStarted","Data":"dc103be63f4e51d4f249faa9ad84486845b06bccae4f96c7661bd979192af702"}
Jul 31 14:51:41.210650 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:41.210624    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"
Jul 31 14:51:41.366892 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:41.366799    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23298d5d-9609-4ccd-91b4-d0957aa5b92f-maas-api-service-ca\") pod \"23298d5d-9609-4ccd-91b4-d0957aa5b92f\" (UID: \"23298d5d-9609-4ccd-91b4-d0957aa5b92f\") "
Jul 31 14:51:41.367187 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:41.367162    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23298d5d-9609-4ccd-91b4-d0957aa5b92f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23298d5d-9609-4ccd-91b4-d0957aa5b92f" (UID: "23298d5d-9609-4ccd-91b4-d0957aa5b92f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:41.467899 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:41.467868    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23298d5d-9609-4ccd-91b4-d0957aa5b92f-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:51:42.087057 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:42.087013    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"
Jul 31 14:51:42.087234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:42.087066    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65" event={"ID":"23298d5d-9609-4ccd-91b4-d0957aa5b92f","Type":"ContainerDied","Data":"dc103be63f4e51d4f249faa9ad84486845b06bccae4f96c7661bd979192af702"}
Jul 31 14:51:42.087234 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:42.087139    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc103be63f4e51d4f249faa9ad84486845b06bccae4f96c7661bd979192af702"
Jul 31 14:51:45.661954 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:45.661915    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"]
Jul 31 14:51:45.663927 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:45.663906    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3187acc6-pmx65"]
Jul 31 14:51:47.580245 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:51:47.580213    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23298d5d-9609-4ccd-91b4-d0957aa5b92f" path="/var/lib/kubelet/pods/23298d5d-9609-4ccd-91b4-d0957aa5b92f/volumes"
Jul 31 14:52:45.508818 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:52:45.508785    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:52:45.517702 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:52:45.517680    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:54:18.306011 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.305978    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"]
Jul 31 14:54:18.306481 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.306415    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23298d5d-9609-4ccd-91b4-d0957aa5b92f" containerName="revoke-keys"
Jul 31 14:54:18.306481 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.306427    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23298d5d-9609-4ccd-91b4-d0957aa5b92f" containerName="revoke-keys"
Jul 31 14:54:18.306556 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.306507    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23298d5d-9609-4ccd-91b4-d0957aa5b92f" containerName="revoke-keys"
Jul 31 14:54:18.309492 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.309477    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"
Jul 31 14:54:18.311939 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.311910    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 14:54:18.316352 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.316327    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"]
Jul 31 14:54:18.427933 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.427899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a056f42f-e78f-47b6-8caa-ab014455ebf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b\" (UID: \"a056f42f-e78f-47b6-8caa-ab014455ebf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"
Jul 31 14:54:18.529049 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.528988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a056f42f-e78f-47b6-8caa-ab014455ebf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b\" (UID: \"a056f42f-e78f-47b6-8caa-ab014455ebf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"
Jul 31 14:54:18.529614 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.529594    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a056f42f-e78f-47b6-8caa-ab014455ebf0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b\" (UID: \"a056f42f-e78f-47b6-8caa-ab014455ebf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"
Jul 31 14:54:18.622665 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.622584    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"
Jul 31 14:54:18.759154 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.759124    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"]
Jul 31 14:54:18.761219 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:54:18.761190    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda056f42f_e78f_47b6_8caa_ab014455ebf0.slice/crio-a09470c9f93d6fe419dc81aa9466ae6bb166822204b94f5e621b934906454770 WatchSource:0}: Error finding container a09470c9f93d6fe419dc81aa9466ae6bb166822204b94f5e621b934906454770: Status 404 returned error can't find the container with id a09470c9f93d6fe419dc81aa9466ae6bb166822204b94f5e621b934906454770
Jul 31 14:54:18.939816 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:18.939789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b" event={"ID":"a056f42f-e78f-47b6-8caa-ab014455ebf0","Type":"ContainerStarted","Data":"a09470c9f93d6fe419dc81aa9466ae6bb166822204b94f5e621b934906454770"}
Jul 31 14:54:19.946272 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:19.946238    2558 generic.go:356] "Generic (PLEG): container finished" podID="a056f42f-e78f-47b6-8caa-ab014455ebf0" containerID="d42eeb13de0606cbc9296bd5507650190b164a9ff9eb8d8cc0dcf57f02bd41c5" exitCode=0
Jul 31 14:54:19.946673 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:19.946331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b" event={"ID":"a056f42f-e78f-47b6-8caa-ab014455ebf0","Type":"ContainerDied","Data":"d42eeb13de0606cbc9296bd5507650190b164a9ff9eb8d8cc0dcf57f02bd41c5"}
Jul 31 14:54:21.084821 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:21.084797    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"
Jul 31 14:54:21.153157 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:21.153116    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a056f42f-e78f-47b6-8caa-ab014455ebf0-maas-api-service-ca\") pod \"a056f42f-e78f-47b6-8caa-ab014455ebf0\" (UID: \"a056f42f-e78f-47b6-8caa-ab014455ebf0\") "
Jul 31 14:54:21.153474 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:21.153450    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a056f42f-e78f-47b6-8caa-ab014455ebf0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a056f42f-e78f-47b6-8caa-ab014455ebf0" (UID: "a056f42f-e78f-47b6-8caa-ab014455ebf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:54:21.254506 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:21.254422    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a056f42f-e78f-47b6-8caa-ab014455ebf0-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:54:21.959632 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:21.959541    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"
Jul 31 14:54:21.959632 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:21.959579    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b" event={"ID":"a056f42f-e78f-47b6-8caa-ab014455ebf0","Type":"ContainerDied","Data":"a09470c9f93d6fe419dc81aa9466ae6bb166822204b94f5e621b934906454770"}
Jul 31 14:54:21.959632 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:21.959627    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a09470c9f93d6fe419dc81aa9466ae6bb166822204b94f5e621b934906454770"
Jul 31 14:54:29.001195 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:29.001159    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"]
Jul 31 14:54:29.003920 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:29.003900    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7f0814aa-tjt7b"]
Jul 31 14:54:29.579368 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:54:29.579332    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a056f42f-e78f-47b6-8caa-ab014455ebf0" path="/var/lib/kubelet/pods/a056f42f-e78f-47b6-8caa-ab014455ebf0/volumes"
Jul 31 14:55:16.335153 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.335120    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"]
Jul 31 14:55:16.335693 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.335383    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs" podUID="82771e8a-ad35-41df-b961-ef816c1fff80" containerName="manager" containerID="cri-o://8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100" gracePeriod=10
Jul 31 14:55:16.594960 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.594904    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:55:16.671430 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.671393    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82771e8a-ad35-41df-b961-ef816c1fff80-extensions-socket-volume\") pod \"82771e8a-ad35-41df-b961-ef816c1fff80\" (UID: \"82771e8a-ad35-41df-b961-ef816c1fff80\") "
Jul 31 14:55:16.671614 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.671504    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf6m9\" (UniqueName: \"kubernetes.io/projected/82771e8a-ad35-41df-b961-ef816c1fff80-kube-api-access-cf6m9\") pod \"82771e8a-ad35-41df-b961-ef816c1fff80\" (UID: \"82771e8a-ad35-41df-b961-ef816c1fff80\") "
Jul 31 14:55:16.671878 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.671846    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82771e8a-ad35-41df-b961-ef816c1fff80-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "82771e8a-ad35-41df-b961-ef816c1fff80" (UID: "82771e8a-ad35-41df-b961-ef816c1fff80"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:55:16.672228 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.672190    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82771e8a-ad35-41df-b961-ef816c1fff80-extensions-socket-volume\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:55:16.673649 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.673627    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82771e8a-ad35-41df-b961-ef816c1fff80-kube-api-access-cf6m9" (OuterVolumeSpecName: "kube-api-access-cf6m9") pod "82771e8a-ad35-41df-b961-ef816c1fff80" (UID: "82771e8a-ad35-41df-b961-ef816c1fff80"). InnerVolumeSpecName "kube-api-access-cf6m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:55:16.773262 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:16.773224    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf6m9\" (UniqueName: \"kubernetes.io/projected/82771e8a-ad35-41df-b961-ef816c1fff80-kube-api-access-cf6m9\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:55:17.271616 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.271576    2558 generic.go:356] "Generic (PLEG): container finished" podID="82771e8a-ad35-41df-b961-ef816c1fff80" containerID="8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100" exitCode=0
Jul 31 14:55:17.271798 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.271651    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"
Jul 31 14:55:17.271798 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.271669    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs" event={"ID":"82771e8a-ad35-41df-b961-ef816c1fff80","Type":"ContainerDied","Data":"8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100"}
Jul 31 14:55:17.271798 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.271751    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs" event={"ID":"82771e8a-ad35-41df-b961-ef816c1fff80","Type":"ContainerDied","Data":"66f7ff4a9ee871104fa18181f8de43692e9865aaa0e49fa8e55e061b8611193f"}
Jul 31 14:55:17.271798 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.271773    2558 scope.go:117] "RemoveContainer" containerID="8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100"
Jul 31 14:55:17.282487 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.282469    2558 scope.go:117] "RemoveContainer" containerID="8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100"
Jul 31 14:55:17.282748 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:55:17.282730    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100\": container with ID starting with 8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100 not found: ID does not exist" containerID="8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100"
Jul 31 14:55:17.282802 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.282757    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100"} err="failed to get container status \"8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100\": rpc error: code = NotFound desc = could not find container \"8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100\": container with ID starting with 8c8b31b41b5d70ea7502e9ca09a2ad272f2adee7c1efaa7689438cbefb61f100 not found: ID does not exist"
Jul 31 14:55:17.294346 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.294325    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"]
Jul 31 14:55:17.297569 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.297548    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-2mlrs"]
Jul 31 14:55:17.579408 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:55:17.579366    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82771e8a-ad35-41df-b961-ef816c1fff80" path="/var/lib/kubelet/pods/82771e8a-ad35-41df-b961-ef816c1fff80/volumes"
Jul 31 14:56:31.959656 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.959612    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"]
Jul 31 14:56:31.960152 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.960075    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82771e8a-ad35-41df-b961-ef816c1fff80" containerName="manager"
Jul 31 14:56:31.960152 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.960089    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="82771e8a-ad35-41df-b961-ef816c1fff80" containerName="manager"
Jul 31 14:56:31.960152 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.960097    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a056f42f-e78f-47b6-8caa-ab014455ebf0" containerName="revoke-keys"
Jul 31 14:56:31.960152 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.960102    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a056f42f-e78f-47b6-8caa-ab014455ebf0" containerName="revoke-keys"
Jul 31 14:56:31.960295 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.960166    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a056f42f-e78f-47b6-8caa-ab014455ebf0" containerName="revoke-keys"
Jul 31 14:56:31.960295 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.960175    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="82771e8a-ad35-41df-b961-ef816c1fff80" containerName="manager"
Jul 31 14:56:31.963301 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.963286    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:31.965906 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.965877    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-q27fh\""
Jul 31 14:56:31.974547 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:31.974522    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"]
Jul 31 14:56:32.040370 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.040338    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f1fcfc-a32b-4f95-837a-e4049c8400c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-7n8ms\" (UID: \"b1f1fcfc-a32b-4f95-837a-e4049c8400c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:32.040534 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.040407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwx8l\" (UniqueName: \"kubernetes.io/projected/b1f1fcfc-a32b-4f95-837a-e4049c8400c4-kube-api-access-dwx8l\") pod \"kuadrant-operator-controller-manager-68cf589589-7n8ms\" (UID: \"b1f1fcfc-a32b-4f95-837a-e4049c8400c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:32.141528 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.141491    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwx8l\" (UniqueName: \"kubernetes.io/projected/b1f1fcfc-a32b-4f95-837a-e4049c8400c4-kube-api-access-dwx8l\") pod \"kuadrant-operator-controller-manager-68cf589589-7n8ms\" (UID: \"b1f1fcfc-a32b-4f95-837a-e4049c8400c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:32.141708 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.141547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f1fcfc-a32b-4f95-837a-e4049c8400c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-7n8ms\" (UID: \"b1f1fcfc-a32b-4f95-837a-e4049c8400c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:32.141930 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.141913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1f1fcfc-a32b-4f95-837a-e4049c8400c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-7n8ms\" (UID: \"b1f1fcfc-a32b-4f95-837a-e4049c8400c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:32.153117 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.153090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwx8l\" (UniqueName: \"kubernetes.io/projected/b1f1fcfc-a32b-4f95-837a-e4049c8400c4-kube-api-access-dwx8l\") pod \"kuadrant-operator-controller-manager-68cf589589-7n8ms\" (UID: \"b1f1fcfc-a32b-4f95-837a-e4049c8400c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:32.273861 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.273764    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:32.410593 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.410566    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"]
Jul 31 14:56:32.412646 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:56:32.412622    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f1fcfc_a32b_4f95_837a_e4049c8400c4.slice/crio-120691f0e360b5ca4a9ceec0da3ace775102729f109b9488dc9784d9ea610449 WatchSource:0}: Error finding container 120691f0e360b5ca4a9ceec0da3ace775102729f109b9488dc9784d9ea610449: Status 404 returned error can't find the container with id 120691f0e360b5ca4a9ceec0da3ace775102729f109b9488dc9784d9ea610449
Jul 31 14:56:32.677836 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.677798    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms" event={"ID":"b1f1fcfc-a32b-4f95-837a-e4049c8400c4","Type":"ContainerStarted","Data":"65dc2af26f1e6df7dbd5b392cf8c583fca218ee1668438b9907f04b361dc23fd"}
Jul 31 14:56:32.678062 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.677857    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms" event={"ID":"b1f1fcfc-a32b-4f95-837a-e4049c8400c4","Type":"ContainerStarted","Data":"120691f0e360b5ca4a9ceec0da3ace775102729f109b9488dc9784d9ea610449"}
Jul 31 14:56:32.678062 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.677979    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:32.698212 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:32.698171    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms" podStartSLOduration=1.698155448 podStartE2EDuration="1.698155448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:56:32.69712348 +0000 UTC m=+2631.569686733" watchObservedRunningTime="2026-07-31 14:56:32.698155448 +0000 UTC m=+2631.570718700"
Jul 31 14:56:43.685389 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:43.685358    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7n8ms"
Jul 31 14:56:45.595546 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:56:45.595514    2558 scope.go:117] "RemoveContainer" containerID="e374f8605042dbc7ae30c5c333787c94d06925bae26ae5fe56098aa8248c241b"
Jul 31 14:57:30.172459 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.172418    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"]
Jul 31 14:57:30.176854 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.176833    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.180701 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.180680    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-86b90147\"/\"default-dockercfg-8zwlf\""
Jul 31 14:57:30.180845 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.180729    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-86b90147\"/\"e2e-lifecycle-model-86b90147-kserve-self-signed-certs\""
Jul 31 14:57:30.180845 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.180808    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-86b90147\"/\"kube-root-ca.crt\""
Jul 31 14:57:30.180845 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.180824    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-86b90147\"/\"openshift-service-ca.crt\""
Jul 31 14:57:30.187313 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.187288    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"]
Jul 31 14:57:30.280653 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.280613    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-home\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.280653 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.280660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9866737-081e-4ed7-b712-da3833c6b03a-tls-certs\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.280890 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.280729    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-model-cache\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.280890 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.280772    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vbl8\" (UniqueName: \"kubernetes.io/projected/f9866737-081e-4ed7-b712-da3833c6b03a-kube-api-access-6vbl8\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.280890 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.280792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-dshm\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.280989 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.280886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-tmp-dir\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382083 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382049    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-tmp-dir\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382272 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382106    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-home\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382272 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9866737-081e-4ed7-b712-da3833c6b03a-tls-certs\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382272 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-model-cache\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382272 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vbl8\" (UniqueName: \"kubernetes.io/projected/f9866737-081e-4ed7-b712-da3833c6b03a-kube-api-access-6vbl8\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382272 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382218    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-dshm\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382547 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382496    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-tmp-dir\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382608 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-home\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.382656 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.382601    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-model-cache\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.384501 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.384478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-dshm\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.384804 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.384785    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9866737-081e-4ed7-b712-da3833c6b03a-tls-certs\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.390413 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.390389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vbl8\" (UniqueName: \"kubernetes.io/projected/f9866737-081e-4ed7-b712-da3833c6b03a-kube-api-access-6vbl8\") pod \"e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") " pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.490046 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.489941    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:30.628903 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.628875    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"]
Jul 31 14:57:30.630848 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:57:30.630816    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9866737_081e_4ed7_b712_da3833c6b03a.slice/crio-89b11790b8b159f69e5fb5195280b5ed84def616e06a937d56bb4c1099488e6a WatchSource:0}: Error finding container 89b11790b8b159f69e5fb5195280b5ed84def616e06a937d56bb4c1099488e6a: Status 404 returned error can't find the container with id 89b11790b8b159f69e5fb5195280b5ed84def616e06a937d56bb4c1099488e6a
Jul 31 14:57:30.998388 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.998352    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb" event={"ID":"f9866737-081e-4ed7-b712-da3833c6b03a","Type":"ContainerStarted","Data":"9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4"}
Jul 31 14:57:30.998388 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:30.998396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb" event={"ID":"f9866737-081e-4ed7-b712-da3833c6b03a","Type":"ContainerStarted","Data":"89b11790b8b159f69e5fb5195280b5ed84def616e06a937d56bb4c1099488e6a"}
Jul 31 14:57:31.016211 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:31.016162    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb" podStartSLOduration=1.01614552 podStartE2EDuration="1.01614552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:31.014234282 +0000 UTC m=+2689.886797535" watchObservedRunningTime="2026-07-31 14:57:31.01614552 +0000 UTC m=+2689.888708773"
Jul 31 14:57:40.490317 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:40.490228    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:40.490317 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:40.490267    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:40.503245 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:40.503221    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:41.071606 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:41.071577    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:57:45.549055 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:45.549007    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:57:45.557332 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:45.557307    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 14:57:45.615434 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:45.615404    2558 scope.go:117] "RemoveContainer" containerID="e20ab8f29400997264bc75d39a89c2a904c6290a7d55f43da77abd7694d5c954"
Jul 31 14:57:58.671179 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:58.671147    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"]
Jul 31 14:57:58.674753 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:58.674730    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"
Jul 31 14:57:58.679094 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:58.677975    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 14:57:58.685116 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:58.685067    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"]
Jul 31 14:57:58.742244 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:58.742205    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2979-157e-4429-aa2f-0b684facbdf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86b90147-xcw8w\" (UID: \"a98b2979-157e-4429-aa2f-0b684facbdf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"
Jul 31 14:57:58.843400 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:58.843369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2979-157e-4429-aa2f-0b684facbdf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86b90147-xcw8w\" (UID: \"a98b2979-157e-4429-aa2f-0b684facbdf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"
Jul 31 14:57:58.843962 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:58.843941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2979-157e-4429-aa2f-0b684facbdf5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-86b90147-xcw8w\" (UID: \"a98b2979-157e-4429-aa2f-0b684facbdf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"
Jul 31 14:57:58.994421 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:58.994345    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"
Jul 31 14:57:59.334423 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:57:59.334398    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"]
Jul 31 14:57:59.336347 ip-10-0-139-143 kubenswrapper[2558]: W0731 14:57:59.336315    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98b2979_157e_4429_aa2f_0b684facbdf5.slice/crio-2a31c924aa8c07dc2b0fe1698df774118780ec10a6b6ad79859c974372e31125 WatchSource:0}: Error finding container 2a31c924aa8c07dc2b0fe1698df774118780ec10a6b6ad79859c974372e31125: Status 404 returned error can't find the container with id 2a31c924aa8c07dc2b0fe1698df774118780ec10a6b6ad79859c974372e31125
Jul 31 14:58:00.162215 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:00.162178    2558 generic.go:356] "Generic (PLEG): container finished" podID="a98b2979-157e-4429-aa2f-0b684facbdf5" containerID="ecf28d1ba631eae2afac04eb689061f6ae91f97648cda0f809b7b392535b8b4b" exitCode=0
Jul 31 14:58:00.162606 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:00.162261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w" event={"ID":"a98b2979-157e-4429-aa2f-0b684facbdf5","Type":"ContainerDied","Data":"ecf28d1ba631eae2afac04eb689061f6ae91f97648cda0f809b7b392535b8b4b"}
Jul 31 14:58:00.162606 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:00.162311    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w" event={"ID":"a98b2979-157e-4429-aa2f-0b684facbdf5","Type":"ContainerStarted","Data":"2a31c924aa8c07dc2b0fe1698df774118780ec10a6b6ad79859c974372e31125"}
Jul 31 14:58:01.308265 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:01.308236    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"
Jul 31 14:58:01.365381 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:01.365355    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2979-157e-4429-aa2f-0b684facbdf5-maas-api-service-ca\") pod \"a98b2979-157e-4429-aa2f-0b684facbdf5\" (UID: \"a98b2979-157e-4429-aa2f-0b684facbdf5\") "
Jul 31 14:58:01.365741 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:01.365705    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a98b2979-157e-4429-aa2f-0b684facbdf5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a98b2979-157e-4429-aa2f-0b684facbdf5" (UID: "a98b2979-157e-4429-aa2f-0b684facbdf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:58:01.466481 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:01.466403    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a98b2979-157e-4429-aa2f-0b684facbdf5-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:58:02.173843 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:02.173810    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"
Jul 31 14:58:02.174040 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:02.173839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w" event={"ID":"a98b2979-157e-4429-aa2f-0b684facbdf5","Type":"ContainerDied","Data":"2a31c924aa8c07dc2b0fe1698df774118780ec10a6b6ad79859c974372e31125"}
Jul 31 14:58:02.174040 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:02.173892    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a31c924aa8c07dc2b0fe1698df774118780ec10a6b6ad79859c974372e31125"
Jul 31 14:58:14.609928 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:14.609893    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"]
Jul 31 14:58:14.614727 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:14.614699    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-86b90147-xcw8w"]
Jul 31 14:58:15.580425 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:15.580391    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a98b2979-157e-4429-aa2f-0b684facbdf5" path="/var/lib/kubelet/pods/a98b2979-157e-4429-aa2f-0b684facbdf5/volumes"
Jul 31 14:58:22.916953 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:22.916918    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"]
Jul 31 14:58:37.936326 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:37.936286    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb" podUID="f9866737-081e-4ed7-b712-da3833c6b03a" containerName="main" containerID="cri-o://9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4" gracePeriod=45
Jul 31 14:58:38.184391 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.184364    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:58:38.211218 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211145    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-tmp-dir\") pod \"f9866737-081e-4ed7-b712-da3833c6b03a\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") "
Jul 31 14:58:38.211218 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211205    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-home\") pod \"f9866737-081e-4ed7-b712-da3833c6b03a\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") "
Jul 31 14:58:38.211466 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211225    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vbl8\" (UniqueName: \"kubernetes.io/projected/f9866737-081e-4ed7-b712-da3833c6b03a-kube-api-access-6vbl8\") pod \"f9866737-081e-4ed7-b712-da3833c6b03a\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") "
Jul 31 14:58:38.211466 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211305    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9866737-081e-4ed7-b712-da3833c6b03a-tls-certs\") pod \"f9866737-081e-4ed7-b712-da3833c6b03a\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") "
Jul 31 14:58:38.211466 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211332    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-model-cache\") pod \"f9866737-081e-4ed7-b712-da3833c6b03a\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") "
Jul 31 14:58:38.211466 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211350    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-dshm\") pod \"f9866737-081e-4ed7-b712-da3833c6b03a\" (UID: \"f9866737-081e-4ed7-b712-da3833c6b03a\") "
Jul 31 14:58:38.211684 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211465    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9866737-081e-4ed7-b712-da3833c6b03a" (UID: "f9866737-081e-4ed7-b712-da3833c6b03a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:58:38.211684 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211601    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:58:38.211914 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.211888    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-home" (OuterVolumeSpecName: "home") pod "f9866737-081e-4ed7-b712-da3833c6b03a" (UID: "f9866737-081e-4ed7-b712-da3833c6b03a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:58:38.212069 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.212047    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9866737-081e-4ed7-b712-da3833c6b03a" (UID: "f9866737-081e-4ed7-b712-da3833c6b03a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:58:38.213928 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.213901    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-dshm" (OuterVolumeSpecName: "dshm") pod "f9866737-081e-4ed7-b712-da3833c6b03a" (UID: "f9866737-081e-4ed7-b712-da3833c6b03a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:58:38.214082 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.213924    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9866737-081e-4ed7-b712-da3833c6b03a-kube-api-access-6vbl8" (OuterVolumeSpecName: "kube-api-access-6vbl8") pod "f9866737-081e-4ed7-b712-da3833c6b03a" (UID: "f9866737-081e-4ed7-b712-da3833c6b03a"). InnerVolumeSpecName "kube-api-access-6vbl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:58:38.214189 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.214168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9866737-081e-4ed7-b712-da3833c6b03a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9866737-081e-4ed7-b712-da3833c6b03a" (UID: "f9866737-081e-4ed7-b712-da3833c6b03a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:58:38.312615 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.312582    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9866737-081e-4ed7-b712-da3833c6b03a-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:58:38.312615 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.312614    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:58:38.312615 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.312623    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:58:38.312878 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.312631    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9866737-081e-4ed7-b712-da3833c6b03a-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:58:38.312878 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.312640    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vbl8\" (UniqueName: \"kubernetes.io/projected/f9866737-081e-4ed7-b712-da3833c6b03a-kube-api-access-6vbl8\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 14:58:38.384111 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.384076    2558 generic.go:356] "Generic (PLEG): container finished" podID="f9866737-081e-4ed7-b712-da3833c6b03a" containerID="9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4" exitCode=0
Jul 31 14:58:38.384252 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.384120    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"
Jul 31 14:58:38.384252 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.384165    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb" event={"ID":"f9866737-081e-4ed7-b712-da3833c6b03a","Type":"ContainerDied","Data":"9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4"}
Jul 31 14:58:38.384252 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.384207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb" event={"ID":"f9866737-081e-4ed7-b712-da3833c6b03a","Type":"ContainerDied","Data":"89b11790b8b159f69e5fb5195280b5ed84def616e06a937d56bb4c1099488e6a"}
Jul 31 14:58:38.384252 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.384235    2558 scope.go:117] "RemoveContainer" containerID="9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4"
Jul 31 14:58:38.393736 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.393718    2558 scope.go:117] "RemoveContainer" containerID="9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4"
Jul 31 14:58:38.393985 ip-10-0-139-143 kubenswrapper[2558]: E0731 14:58:38.393967    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4\": container with ID starting with 9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4 not found: ID does not exist" containerID="9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4"
Jul 31 14:58:38.394071 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.393994    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4"} err="failed to get container status \"9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4\": rpc error: code = NotFound desc = could not find container \"9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4\": container with ID starting with 9fbb81b969d8b963027e3c4eb95c278a713ba42e7a61f354f980fb1cedc5bbb4 not found: ID does not exist"
Jul 31 14:58:38.405208 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.405186    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"]
Jul 31 14:58:38.408645 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:38.408626    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-86b90147/e2e-lifecycle-model-86b90147-kserve-85bfcc4c5f-ppqbb"]
Jul 31 14:58:39.579784 ip-10-0-139-143 kubenswrapper[2558]: I0731 14:58:39.579750    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9866737-081e-4ed7-b712-da3833c6b03a" path="/var/lib/kubelet/pods/f9866737-081e-4ed7-b712-da3833c6b03a/volumes"
Jul 31 15:00:45.654912 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:00:45.654836    2558 scope.go:117] "RemoveContainer" containerID="d42eeb13de0606cbc9296bd5507650190b164a9ff9eb8d8cc0dcf57f02bd41c5"
Jul 31 15:01:16.474714 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.474678    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"]
Jul 31 15:01:16.475319 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.475299    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9866737-081e-4ed7-b712-da3833c6b03a" containerName="main"
Jul 31 15:01:16.475409 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.475322    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9866737-081e-4ed7-b712-da3833c6b03a" containerName="main"
Jul 31 15:01:16.475409 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.475335    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a98b2979-157e-4429-aa2f-0b684facbdf5" containerName="revoke-keys"
Jul 31 15:01:16.475409 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.475343    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a98b2979-157e-4429-aa2f-0b684facbdf5" containerName="revoke-keys"
Jul 31 15:01:16.475567 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.475467    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a98b2979-157e-4429-aa2f-0b684facbdf5" containerName="revoke-keys"
Jul 31 15:01:16.475567 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.475486    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9866737-081e-4ed7-b712-da3833c6b03a" containerName="main"
Jul 31 15:01:16.478621 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.478601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.481481 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.481458    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-55959d\"/\"default-dockercfg-mmxfc\""
Jul 31 15:01:16.481586 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.481491    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-55959d\"/\"openshift-service-ca.crt\""
Jul 31 15:01:16.483060 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.483015    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-55959d\"/\"kube-root-ca.crt\""
Jul 31 15:01:16.483160 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.483098    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-55959d\"/\"test-model-55959d-kserve-self-signed-certs\""
Jul 31 15:01:16.486698 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.486676    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"]
Jul 31 15:01:16.564663 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.564624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-dshm\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.564844 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.564707    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5vw\" (UniqueName: \"kubernetes.io/projected/66468b41-db22-438a-9bb7-9e058d5adbcf-kube-api-access-ms5vw\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.564844 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.564749    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-home\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.564844 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.564774    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-model-cache\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.564844 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.564824    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468b41-db22-438a-9bb7-9e058d5adbcf-tls-certs\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.565114 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.564876    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-tmp-dir\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.665905 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.665872    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-home\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.666137 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.665912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-model-cache\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.666137 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.665969    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468b41-db22-438a-9bb7-9e058d5adbcf-tls-certs\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.666137 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.666001    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-tmp-dir\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.666137 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.666083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-dshm\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.666379 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.666166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5vw\" (UniqueName: \"kubernetes.io/projected/66468b41-db22-438a-9bb7-9e058d5adbcf-kube-api-access-ms5vw\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.666379 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.666362    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-home\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.666480 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.666425    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-model-cache\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.666480 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.666451    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-tmp-dir\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.668502 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.668480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-dshm\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.668781 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.668763    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468b41-db22-438a-9bb7-9e058d5adbcf-tls-certs\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.673676 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.673658    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5vw\" (UniqueName: \"kubernetes.io/projected/66468b41-db22-438a-9bb7-9e058d5adbcf-kube-api-access-ms5vw\") pod \"test-model-55959d-kserve-79b74cd59-4bl2z\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") " pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.793044 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.792994    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:16.939765 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:16.939735    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"]
Jul 31 15:01:16.941109 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:01:16.941082    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66468b41_db22_438a_9bb7_9e058d5adbcf.slice/crio-734b190551e98bb3e6a940a0bc422999689f6ce014fb2ce91e3f08e184427191 WatchSource:0}: Error finding container 734b190551e98bb3e6a940a0bc422999689f6ce014fb2ce91e3f08e184427191: Status 404 returned error can't find the container with id 734b190551e98bb3e6a940a0bc422999689f6ce014fb2ce91e3f08e184427191
Jul 31 15:01:17.269625 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:17.269518    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z" event={"ID":"66468b41-db22-438a-9bb7-9e058d5adbcf","Type":"ContainerStarted","Data":"76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed"}
Jul 31 15:01:17.269625 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:17.269576    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z" event={"ID":"66468b41-db22-438a-9bb7-9e058d5adbcf","Type":"ContainerStarted","Data":"734b190551e98bb3e6a940a0bc422999689f6ce014fb2ce91e3f08e184427191"}
Jul 31 15:01:17.287314 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:17.287264    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z" podStartSLOduration=1.28724843 podStartE2EDuration="1.28724843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:17.285520524 +0000 UTC m=+2916.158083787" watchObservedRunningTime="2026-07-31 15:01:17.28724843 +0000 UTC m=+2916.159811683"
Jul 31 15:01:26.794054 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:26.794006    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:26.794565 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:26.794094    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:26.806353 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:26.806328    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:27.337655 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:27.337627    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:01:42.180690 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.180659    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"]
Jul 31 15:01:42.184521 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.184501    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.187054 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.187012    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef4b6d\"/\"openshift-service-ca.crt\""
Jul 31 15:01:42.188247 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.188224    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef4b6d\"/\"test-model-ef4b6d-kserve-self-signed-certs\""
Jul 31 15:01:42.188321 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.188249    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef4b6d\"/\"default-dockercfg-bz75h\""
Jul 31 15:01:42.188378 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.188340    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef4b6d\"/\"kube-root-ca.crt\""
Jul 31 15:01:42.194526 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.194505    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"]
Jul 31 15:01:42.305412 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.305372    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-model-cache\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.305602 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.305419    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tls-certs\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.305602 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.305536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvfd\" (UniqueName: \"kubernetes.io/projected/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-kube-api-access-cdvfd\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.305602 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.305580    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-dshm\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.305602 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.305601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tmp-dir\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.305770 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.305670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-home\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406216 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406183    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-model-cache\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406421 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406225    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tls-certs\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406421 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvfd\" (UniqueName: \"kubernetes.io/projected/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-kube-api-access-cdvfd\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406421 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406319    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-dshm\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406421 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406349    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tmp-dir\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406421 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406404    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-home\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406685 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-model-cache\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406730 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tmp-dir\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.406775 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.406759    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-home\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.408538 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.408517    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-dshm\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.408874 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.408855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tls-certs\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.415331 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.415310    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvfd\" (UniqueName: \"kubernetes.io/projected/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-kube-api-access-cdvfd\") pod \"test-model-ef4b6d-kserve-86d9fdb555-6m744\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") " pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.495426 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.495335    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:42.633307 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:42.633282    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"]
Jul 31 15:01:42.635486 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:01:42.635464    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5e1ac7_314b_4633_9b7b_9366c7850ddb.slice/crio-7507a305e42d72554bcacef0a6a99b31048ae755a1deb8373fffd8537ba6bc6a WatchSource:0}: Error finding container 7507a305e42d72554bcacef0a6a99b31048ae755a1deb8373fffd8537ba6bc6a: Status 404 returned error can't find the container with id 7507a305e42d72554bcacef0a6a99b31048ae755a1deb8373fffd8537ba6bc6a
Jul 31 15:01:43.411814 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:43.411774    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744" event={"ID":"ca5e1ac7-314b-4633-9b7b-9366c7850ddb","Type":"ContainerStarted","Data":"9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa"}
Jul 31 15:01:43.412323 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:43.411822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744" event={"ID":"ca5e1ac7-314b-4633-9b7b-9366c7850ddb","Type":"ContainerStarted","Data":"7507a305e42d72554bcacef0a6a99b31048ae755a1deb8373fffd8537ba6bc6a"}
Jul 31 15:01:43.429410 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:43.429365    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744" podStartSLOduration=1.429349767 podStartE2EDuration="1.429349767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:01:43.428209602 +0000 UTC m=+2942.300772856" watchObservedRunningTime="2026-07-31 15:01:43.429349767 +0000 UTC m=+2942.301913020"
Jul 31 15:01:52.495500 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:52.495462    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:52.496087 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:52.495545    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:52.509187 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:52.509153    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:01:53.478393 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:01:53.478315    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:02:26.926954 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:26.926911    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"]
Jul 31 15:02:28.082141 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.082103    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"]
Jul 31 15:02:28.241966 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.241931    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"]
Jul 31 15:02:28.245563 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.245546    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"
Jul 31 15:02:28.248042 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.247999    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:02:28.252395 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.252374    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"]
Jul 31 15:02:28.333688 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.333601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222deee8-8747-40e4-93ae-27678b2ca78f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-55959d-lsc64\" (UID: \"222deee8-8747-40e4-93ae-27678b2ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"
Jul 31 15:02:28.434359 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.434322    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222deee8-8747-40e4-93ae-27678b2ca78f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-55959d-lsc64\" (UID: \"222deee8-8747-40e4-93ae-27678b2ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"
Jul 31 15:02:28.434948 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.434929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222deee8-8747-40e4-93ae-27678b2ca78f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-55959d-lsc64\" (UID: \"222deee8-8747-40e4-93ae-27678b2ca78f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"
Jul 31 15:02:28.558077 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.558045    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"
Jul 31 15:02:28.694231 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:28.694203    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"]
Jul 31 15:02:28.696202 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:02:28.696165    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222deee8_8747_40e4_93ae_27678b2ca78f.slice/crio-6d43d6084685366ea07aaf82dd66861c2634ab1e83ab2d5d7eae62104c2a02f9 WatchSource:0}: Error finding container 6d43d6084685366ea07aaf82dd66861c2634ab1e83ab2d5d7eae62104c2a02f9: Status 404 returned error can't find the container with id 6d43d6084685366ea07aaf82dd66861c2634ab1e83ab2d5d7eae62104c2a02f9
Jul 31 15:02:29.660427 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:29.660389    2558 generic.go:356] "Generic (PLEG): container finished" podID="222deee8-8747-40e4-93ae-27678b2ca78f" containerID="b320c35e6049c2c048dde2246df1935372494cd1b9ae20d2834334e5cac49ccb" exitCode=0
Jul 31 15:02:29.660868 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:29.660431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64" event={"ID":"222deee8-8747-40e4-93ae-27678b2ca78f","Type":"ContainerDied","Data":"b320c35e6049c2c048dde2246df1935372494cd1b9ae20d2834334e5cac49ccb"}
Jul 31 15:02:29.660868 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:29.660484    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64" event={"ID":"222deee8-8747-40e4-93ae-27678b2ca78f","Type":"ContainerStarted","Data":"6d43d6084685366ea07aaf82dd66861c2634ab1e83ab2d5d7eae62104c2a02f9"}
Jul 31 15:02:30.806585 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:30.806563    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"
Jul 31 15:02:30.957202 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:30.957108    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222deee8-8747-40e4-93ae-27678b2ca78f-maas-api-service-ca\") pod \"222deee8-8747-40e4-93ae-27678b2ca78f\" (UID: \"222deee8-8747-40e4-93ae-27678b2ca78f\") "
Jul 31 15:02:30.957534 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:30.957507    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222deee8-8747-40e4-93ae-27678b2ca78f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "222deee8-8747-40e4-93ae-27678b2ca78f" (UID: "222deee8-8747-40e4-93ae-27678b2ca78f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:02:31.058043 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:31.058004    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222deee8-8747-40e4-93ae-27678b2ca78f-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:31.679709 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:31.679672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64" event={"ID":"222deee8-8747-40e4-93ae-27678b2ca78f","Type":"ContainerDied","Data":"6d43d6084685366ea07aaf82dd66861c2634ab1e83ab2d5d7eae62104c2a02f9"}
Jul 31 15:02:31.679880 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:31.679750    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d43d6084685366ea07aaf82dd66861c2634ab1e83ab2d5d7eae62104c2a02f9"
Jul 31 15:02:31.679880 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:31.679691    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"
Jul 31 15:02:38.993619 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:38.993583    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"]
Jul 31 15:02:38.997785 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:38.997753    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-55959d-lsc64"]
Jul 31 15:02:39.579861 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:39.579817    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="222deee8-8747-40e4-93ae-27678b2ca78f" path="/var/lib/kubelet/pods/222deee8-8747-40e4-93ae-27678b2ca78f/volumes"
Jul 31 15:02:41.946767 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:41.946706    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z" podUID="66468b41-db22-438a-9bb7-9e058d5adbcf" containerName="main" containerID="cri-o://76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed" gracePeriod=45
Jul 31 15:02:42.204982 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.204920    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:02:42.261801 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.261769    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468b41-db22-438a-9bb7-9e058d5adbcf-tls-certs\") pod \"66468b41-db22-438a-9bb7-9e058d5adbcf\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") "
Jul 31 15:02:42.261985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.261843    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-tmp-dir\") pod \"66468b41-db22-438a-9bb7-9e058d5adbcf\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") "
Jul 31 15:02:42.261985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.261860    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-home\") pod \"66468b41-db22-438a-9bb7-9e058d5adbcf\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") "
Jul 31 15:02:42.261985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.261882    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms5vw\" (UniqueName: \"kubernetes.io/projected/66468b41-db22-438a-9bb7-9e058d5adbcf-kube-api-access-ms5vw\") pod \"66468b41-db22-438a-9bb7-9e058d5adbcf\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") "
Jul 31 15:02:42.261985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.261919    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-dshm\") pod \"66468b41-db22-438a-9bb7-9e058d5adbcf\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") "
Jul 31 15:02:42.261985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.261957    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-model-cache\") pod \"66468b41-db22-438a-9bb7-9e058d5adbcf\" (UID: \"66468b41-db22-438a-9bb7-9e058d5adbcf\") "
Jul 31 15:02:42.262292 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.262177    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66468b41-db22-438a-9bb7-9e058d5adbcf" (UID: "66468b41-db22-438a-9bb7-9e058d5adbcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:42.262292 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.262194    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-home" (OuterVolumeSpecName: "home") pod "66468b41-db22-438a-9bb7-9e058d5adbcf" (UID: "66468b41-db22-438a-9bb7-9e058d5adbcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:42.262292 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.262277    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "66468b41-db22-438a-9bb7-9e058d5adbcf" (UID: "66468b41-db22-438a-9bb7-9e058d5adbcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:42.262400 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.262334    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:42.262400 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.262348    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:42.264103 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.264079    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-dshm" (OuterVolumeSpecName: "dshm") pod "66468b41-db22-438a-9bb7-9e058d5adbcf" (UID: "66468b41-db22-438a-9bb7-9e058d5adbcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:42.264209 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.264146    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66468b41-db22-438a-9bb7-9e058d5adbcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66468b41-db22-438a-9bb7-9e058d5adbcf" (UID: "66468b41-db22-438a-9bb7-9e058d5adbcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:02:42.264209 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.264164    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66468b41-db22-438a-9bb7-9e058d5adbcf-kube-api-access-ms5vw" (OuterVolumeSpecName: "kube-api-access-ms5vw") pod "66468b41-db22-438a-9bb7-9e058d5adbcf" (UID: "66468b41-db22-438a-9bb7-9e058d5adbcf"). InnerVolumeSpecName "kube-api-access-ms5vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:02:42.363115 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.363091    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468b41-db22-438a-9bb7-9e058d5adbcf-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:42.363115 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.363113    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms5vw\" (UniqueName: \"kubernetes.io/projected/66468b41-db22-438a-9bb7-9e058d5adbcf-kube-api-access-ms5vw\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:42.363115 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.363122    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:42.363324 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.363133    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468b41-db22-438a-9bb7-9e058d5adbcf-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:42.744570 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.744535    2558 generic.go:356] "Generic (PLEG): container finished" podID="66468b41-db22-438a-9bb7-9e058d5adbcf" containerID="76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed" exitCode=0
Jul 31 15:02:42.744773 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.744579    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"
Jul 31 15:02:42.744773 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.744615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z" event={"ID":"66468b41-db22-438a-9bb7-9e058d5adbcf","Type":"ContainerDied","Data":"76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed"}
Jul 31 15:02:42.744773 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.744668    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z" event={"ID":"66468b41-db22-438a-9bb7-9e058d5adbcf","Type":"ContainerDied","Data":"734b190551e98bb3e6a940a0bc422999689f6ce014fb2ce91e3f08e184427191"}
Jul 31 15:02:42.744773 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.744691    2558 scope.go:117] "RemoveContainer" containerID="76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed"
Jul 31 15:02:42.754865 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.754849    2558 scope.go:117] "RemoveContainer" containerID="76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed"
Jul 31 15:02:42.755132 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:02:42.755115    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed\": container with ID starting with 76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed not found: ID does not exist" containerID="76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed"
Jul 31 15:02:42.755194 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.755139    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed"} err="failed to get container status \"76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed\": rpc error: code = NotFound desc = could not find container \"76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed\": container with ID starting with 76a17161bdb3809208ff462497535af0a4e4ee6cdd06350c53cd2184d9f55bed not found: ID does not exist"
Jul 31 15:02:42.770928 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.770905    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"]
Jul 31 15:02:42.776410 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:42.776381    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-55959d/test-model-55959d-kserve-79b74cd59-4bl2z"]
Jul 31 15:02:43.102196 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.102155    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744" podUID="ca5e1ac7-314b-4633-9b7b-9366c7850ddb" containerName="main" containerID="cri-o://9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa" gracePeriod=45
Jul 31 15:02:43.354571 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.354509    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:02:43.372016 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.371988    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tls-certs\") pod \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") "
Jul 31 15:02:43.372167 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.372155    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-model-cache\") pod \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") "
Jul 31 15:02:43.372221 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.372193    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdvfd\" (UniqueName: \"kubernetes.io/projected/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-kube-api-access-cdvfd\") pod \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") "
Jul 31 15:02:43.372284 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.372251    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tmp-dir\") pod \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") "
Jul 31 15:02:43.372284 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.372278    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-dshm\") pod \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") "
Jul 31 15:02:43.372388 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.372320    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-home\") pod \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\" (UID: \"ca5e1ac7-314b-4633-9b7b-9366c7850ddb\") "
Jul 31 15:02:43.372695 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.372666    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca5e1ac7-314b-4633-9b7b-9366c7850ddb" (UID: "ca5e1ac7-314b-4633-9b7b-9366c7850ddb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:43.373175 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.373137    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca5e1ac7-314b-4633-9b7b-9366c7850ddb" (UID: "ca5e1ac7-314b-4633-9b7b-9366c7850ddb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:43.373286 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.372854    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-home" (OuterVolumeSpecName: "home") pod "ca5e1ac7-314b-4633-9b7b-9366c7850ddb" (UID: "ca5e1ac7-314b-4633-9b7b-9366c7850ddb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:43.373918 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.373515    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.373918 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.373540    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.373918 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.373553    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.375084 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.375057    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-kube-api-access-cdvfd" (OuterVolumeSpecName: "kube-api-access-cdvfd") pod "ca5e1ac7-314b-4633-9b7b-9366c7850ddb" (UID: "ca5e1ac7-314b-4633-9b7b-9366c7850ddb"). InnerVolumeSpecName "kube-api-access-cdvfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:02:43.375195 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.375108    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca5e1ac7-314b-4633-9b7b-9366c7850ddb" (UID: "ca5e1ac7-314b-4633-9b7b-9366c7850ddb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:02:43.375281 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.375263    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-dshm" (OuterVolumeSpecName: "dshm") pod "ca5e1ac7-314b-4633-9b7b-9366c7850ddb" (UID: "ca5e1ac7-314b-4633-9b7b-9366c7850ddb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:02:43.474248 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.474214    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdvfd\" (UniqueName: \"kubernetes.io/projected/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-kube-api-access-cdvfd\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.474248 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.474243    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.474248 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.474252    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca5e1ac7-314b-4633-9b7b-9366c7850ddb-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:43.580110 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.580077    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66468b41-db22-438a-9bb7-9e058d5adbcf" path="/var/lib/kubelet/pods/66468b41-db22-438a-9bb7-9e058d5adbcf/volumes"
Jul 31 15:02:43.753546 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.753460    2558 generic.go:356] "Generic (PLEG): container finished" podID="ca5e1ac7-314b-4633-9b7b-9366c7850ddb" containerID="9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa" exitCode=0
Jul 31 15:02:43.753546 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.753505    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"
Jul 31 15:02:43.753546 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.753541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744" event={"ID":"ca5e1ac7-314b-4633-9b7b-9366c7850ddb","Type":"ContainerDied","Data":"9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa"}
Jul 31 15:02:43.753850 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.753574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744" event={"ID":"ca5e1ac7-314b-4633-9b7b-9366c7850ddb","Type":"ContainerDied","Data":"7507a305e42d72554bcacef0a6a99b31048ae755a1deb8373fffd8537ba6bc6a"}
Jul 31 15:02:43.753850 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.753593    2558 scope.go:117] "RemoveContainer" containerID="9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa"
Jul 31 15:02:43.762590 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.762573    2558 scope.go:117] "RemoveContainer" containerID="9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa"
Jul 31 15:02:43.762844 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:02:43.762825    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa\": container with ID starting with 9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa not found: ID does not exist" containerID="9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa"
Jul 31 15:02:43.762894 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.762853    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa"} err="failed to get container status \"9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa\": rpc error: code = NotFound desc = could not find container \"9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa\": container with ID starting with 9ff036e904510828fddca90fde7dc17daca83740113351400cd7d3756a3ff1fa not found: ID does not exist"
Jul 31 15:02:43.769582 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.769560    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"]
Jul 31 15:02:43.773320 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:43.773298    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ef4b6d/test-model-ef4b6d-kserve-86d9fdb555-6m744"]
Jul 31 15:02:45.580547 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.580513    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca5e1ac7-314b-4633-9b7b-9366c7850ddb" path="/var/lib/kubelet/pods/ca5e1ac7-314b-4633-9b7b-9366c7850ddb/volumes"
Jul 31 15:02:45.582912 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.582887    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:02:45.591474 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.591451    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:02:45.806864 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.806825    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"]
Jul 31 15:02:45.807341 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807322    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca5e1ac7-314b-4633-9b7b-9366c7850ddb" containerName="main"
Jul 31 15:02:45.807341 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807342    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca5e1ac7-314b-4633-9b7b-9366c7850ddb" containerName="main"
Jul 31 15:02:45.807491 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807355    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222deee8-8747-40e4-93ae-27678b2ca78f" containerName="revoke-keys"
Jul 31 15:02:45.807491 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807364    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="222deee8-8747-40e4-93ae-27678b2ca78f" containerName="revoke-keys"
Jul 31 15:02:45.807491 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807399    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66468b41-db22-438a-9bb7-9e058d5adbcf" containerName="main"
Jul 31 15:02:45.807491 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807408    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="66468b41-db22-438a-9bb7-9e058d5adbcf" containerName="main"
Jul 31 15:02:45.807752 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807503    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="222deee8-8747-40e4-93ae-27678b2ca78f" containerName="revoke-keys"
Jul 31 15:02:45.807752 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807518    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="66468b41-db22-438a-9bb7-9e058d5adbcf" containerName="main"
Jul 31 15:02:45.807752 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.807532    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca5e1ac7-314b-4633-9b7b-9366c7850ddb" containerName="main"
Jul 31 15:02:45.812273 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.812253    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"
Jul 31 15:02:45.815311 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.815287    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:02:45.817825 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.817801    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"]
Jul 31 15:02:45.894630 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.894549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8d2a8a-a7ad-4638-9f68-875ee54d1153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4\" (UID: \"0c8d2a8a-a7ad-4638-9f68-875ee54d1153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"
Jul 31 15:02:45.995388 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.995350    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8d2a8a-a7ad-4638-9f68-875ee54d1153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4\" (UID: \"0c8d2a8a-a7ad-4638-9f68-875ee54d1153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"
Jul 31 15:02:45.995989 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:45.995970    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8d2a8a-a7ad-4638-9f68-875ee54d1153-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4\" (UID: \"0c8d2a8a-a7ad-4638-9f68-875ee54d1153\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"
Jul 31 15:02:46.125974 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:46.125940    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"
Jul 31 15:02:46.265557 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:46.265522    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"]
Jul 31 15:02:46.266980 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:02:46.266955    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c8d2a8a_a7ad_4638_9f68_875ee54d1153.slice/crio-e10383e8a9a2baee9e62ad936130d46fbc0df998c7bff765ad448adefdede851 WatchSource:0}: Error finding container e10383e8a9a2baee9e62ad936130d46fbc0df998c7bff765ad448adefdede851: Status 404 returned error can't find the container with id e10383e8a9a2baee9e62ad936130d46fbc0df998c7bff765ad448adefdede851
Jul 31 15:02:46.772818 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:46.772730    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4" event={"ID":"0c8d2a8a-a7ad-4638-9f68-875ee54d1153","Type":"ContainerStarted","Data":"b5e2802b4890e5c67b1c1ba101da8abb37728727dfb8750745cba9c520121f0f"}
Jul 31 15:02:46.772818 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:46.772780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4" event={"ID":"0c8d2a8a-a7ad-4638-9f68-875ee54d1153","Type":"ContainerStarted","Data":"e10383e8a9a2baee9e62ad936130d46fbc0df998c7bff765ad448adefdede851"}
Jul 31 15:02:46.788890 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:46.788844    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4" podStartSLOduration=1.788828835 podStartE2EDuration="1.788828835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:02:46.787001918 +0000 UTC m=+3005.659565171" watchObservedRunningTime="2026-07-31 15:02:46.788828835 +0000 UTC m=+3005.661392087"
Jul 31 15:02:47.779394 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:47.779360    2558 generic.go:356] "Generic (PLEG): container finished" podID="0c8d2a8a-a7ad-4638-9f68-875ee54d1153" containerID="b5e2802b4890e5c67b1c1ba101da8abb37728727dfb8750745cba9c520121f0f" exitCode=0
Jul 31 15:02:47.779832 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:47.779446    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4" event={"ID":"0c8d2a8a-a7ad-4638-9f68-875ee54d1153","Type":"ContainerDied","Data":"b5e2802b4890e5c67b1c1ba101da8abb37728727dfb8750745cba9c520121f0f"}
Jul 31 15:02:48.920567 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:48.920542    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"
Jul 31 15:02:49.023834 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:49.023802    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8d2a8a-a7ad-4638-9f68-875ee54d1153-maas-api-service-ca\") pod \"0c8d2a8a-a7ad-4638-9f68-875ee54d1153\" (UID: \"0c8d2a8a-a7ad-4638-9f68-875ee54d1153\") "
Jul 31 15:02:49.024242 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:49.024220    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c8d2a8a-a7ad-4638-9f68-875ee54d1153-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c8d2a8a-a7ad-4638-9f68-875ee54d1153" (UID: "0c8d2a8a-a7ad-4638-9f68-875ee54d1153"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:02:49.125693 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:49.125661    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c8d2a8a-a7ad-4638-9f68-875ee54d1153-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:02:49.790742 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:49.790716    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"
Jul 31 15:02:49.790918 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:49.790736    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4" event={"ID":"0c8d2a8a-a7ad-4638-9f68-875ee54d1153","Type":"ContainerDied","Data":"e10383e8a9a2baee9e62ad936130d46fbc0df998c7bff765ad448adefdede851"}
Jul 31 15:02:49.790918 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:49.790794    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e10383e8a9a2baee9e62ad936130d46fbc0df998c7bff765ad448adefdede851"
Jul 31 15:02:56.308712 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:56.308678    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"]
Jul 31 15:02:56.314438 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:56.314412    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef4b6d-phgn4"]
Jul 31 15:02:57.579938 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:02:57.579904    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c8d2a8a-a7ad-4638-9f68-875ee54d1153" path="/var/lib/kubelet/pods/0c8d2a8a-a7ad-4638-9f68-875ee54d1153/volumes"
Jul 31 15:03:36.918181 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.918085    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"]
Jul 31 15:03:36.918762 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.918714    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c8d2a8a-a7ad-4638-9f68-875ee54d1153" containerName="revoke-keys"
Jul 31 15:03:36.918762 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.918736    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c8d2a8a-a7ad-4638-9f68-875ee54d1153" containerName="revoke-keys"
Jul 31 15:03:36.918889 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.918870    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c8d2a8a-a7ad-4638-9f68-875ee54d1153" containerName="revoke-keys"
Jul 31 15:03:36.922074 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.922048    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:36.924642 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.924622    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cl2ws\""
Jul 31 15:03:36.924753 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.924622    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2eeeef-serving-cert\""
Jul 31 15:03:36.932654 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.932629    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"]
Jul 31 15:03:36.971192 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.971156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:36.971192 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.971195    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8cjv\" (UniqueName: \"kubernetes.io/projected/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-kube-api-access-m8cjv\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:36.971366 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:36.971216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.071991 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.071955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.071991 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.071992    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8cjv\" (UniqueName: \"kubernetes.io/projected/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-kube-api-access-m8cjv\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.072269 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.072012    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.072269 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:03:37.072130    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-serving-cert: secret "maas-api-e2e-api-b-2eeeef-serving-cert" not found
Jul 31 15:03:37.072269 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:03:37.072232    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls podName:bf6be5b9-bf5f-45a0-9fca-f20659f274ba nodeName:}" failed. No retries permitted until 2026-07-31 15:03:37.572212589 +0000 UTC m=+3056.444775832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls") pod "maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft" (UID: "bf6be5b9-bf5f-45a0-9fca-f20659f274ba") : secret "maas-api-e2e-api-b-2eeeef-serving-cert" not found
Jul 31 15:03:37.072888 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.072862    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-openshift-service-ca\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.083224 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.083199    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8cjv\" (UniqueName: \"kubernetes.io/projected/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-kube-api-access-m8cjv\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.576749 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.576716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.579430 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.579401    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls\") pod \"maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.844700 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.844611    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:37.983461 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.983427    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"]
Jul 31 15:03:37.983998 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:03:37.983970    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6be5b9_bf5f_45a0_9fca_f20659f274ba.slice/crio-f8928809da190241b42e18c16e0472648f1579f686e8d608cda32e4884e2861d WatchSource:0}: Error finding container f8928809da190241b42e18c16e0472648f1579f686e8d608cda32e4884e2861d: Status 404 returned error can't find the container with id f8928809da190241b42e18c16e0472648f1579f686e8d608cda32e4884e2861d
Jul 31 15:03:37.985541 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:37.985517    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:03:38.055551 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:38.055516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft" event={"ID":"bf6be5b9-bf5f-45a0-9fca-f20659f274ba","Type":"ContainerStarted","Data":"f8928809da190241b42e18c16e0472648f1579f686e8d608cda32e4884e2861d"}
Jul 31 15:03:41.075586 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:41.075546    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft" event={"ID":"bf6be5b9-bf5f-45a0-9fca-f20659f274ba","Type":"ContainerStarted","Data":"330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf"}
Jul 31 15:03:41.076045 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:41.075628    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:03:41.094093 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:41.094043    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft" podStartSLOduration=2.907714411 podStartE2EDuration="5.094010458s" totalImagesPullingTime="2.186296047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:41.092670042 +0000 UTC m=+3059.965233310" watchObservedRunningTime="2026-07-31 15:03:41.094010458 +0000 UTC m=+3059.966573711"
Jul 31 15:03:47.092704 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:03:47.092672    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:04:03.932543 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:03.932507    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"]
Jul 31 15:04:03.935197 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:03.935182    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"
Jul 31 15:04:03.937779 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:03.937759    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:04:03.943354 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:03.942832    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"]
Jul 31 15:04:04.032574 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:04.032538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba36e8a4-f319-47b1-981c-07754192b8db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-700618-gk8mn\" (UID: \"ba36e8a4-f319-47b1-981c-07754192b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"
Jul 31 15:04:04.133971 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:04.133937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba36e8a4-f319-47b1-981c-07754192b8db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-700618-gk8mn\" (UID: \"ba36e8a4-f319-47b1-981c-07754192b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"
Jul 31 15:04:04.134657 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:04.134629    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba36e8a4-f319-47b1-981c-07754192b8db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-700618-gk8mn\" (UID: \"ba36e8a4-f319-47b1-981c-07754192b8db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"
Jul 31 15:04:04.248498 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:04.248397    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"
Jul 31 15:04:04.595335 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:04.595310    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"]
Jul 31 15:04:04.597180 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:04:04.597145    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba36e8a4_f319_47b1_981c_07754192b8db.slice/crio-8f1643c457603c50c1fff15fcf62c64faa606d05f3c86d747133009c63d2985f WatchSource:0}: Error finding container 8f1643c457603c50c1fff15fcf62c64faa606d05f3c86d747133009c63d2985f: Status 404 returned error can't find the container with id 8f1643c457603c50c1fff15fcf62c64faa606d05f3c86d747133009c63d2985f
Jul 31 15:04:05.214733 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:05.214641    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn" event={"ID":"ba36e8a4-f319-47b1-981c-07754192b8db","Type":"ContainerStarted","Data":"6f04ad6ebe4be4a820e0a6b1d9562b7ba91b37346cab8f0f557f3ea784a7a4e6"}
Jul 31 15:04:05.214733 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:05.214703    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn" event={"ID":"ba36e8a4-f319-47b1-981c-07754192b8db","Type":"ContainerStarted","Data":"8f1643c457603c50c1fff15fcf62c64faa606d05f3c86d747133009c63d2985f"}
Jul 31 15:04:05.232725 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:05.232675    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn" podStartSLOduration=2.23265847 podStartE2EDuration="2.23265847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:05.230299482 +0000 UTC m=+3084.102862735" watchObservedRunningTime="2026-07-31 15:04:05.23265847 +0000 UTC m=+3084.105221724"
Jul 31 15:04:06.222179 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:06.222140    2558 generic.go:356] "Generic (PLEG): container finished" podID="ba36e8a4-f319-47b1-981c-07754192b8db" containerID="6f04ad6ebe4be4a820e0a6b1d9562b7ba91b37346cab8f0f557f3ea784a7a4e6" exitCode=0
Jul 31 15:04:06.222576 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:06.222186    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn" event={"ID":"ba36e8a4-f319-47b1-981c-07754192b8db","Type":"ContainerDied","Data":"6f04ad6ebe4be4a820e0a6b1d9562b7ba91b37346cab8f0f557f3ea784a7a4e6"}
Jul 31 15:04:07.356238 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:07.356209    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"
Jul 31 15:04:07.462735 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:07.462705    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba36e8a4-f319-47b1-981c-07754192b8db-maas-api-service-ca\") pod \"ba36e8a4-f319-47b1-981c-07754192b8db\" (UID: \"ba36e8a4-f319-47b1-981c-07754192b8db\") "
Jul 31 15:04:07.463105 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:07.463083    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba36e8a4-f319-47b1-981c-07754192b8db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba36e8a4-f319-47b1-981c-07754192b8db" (UID: "ba36e8a4-f319-47b1-981c-07754192b8db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:07.563927 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:07.563896    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba36e8a4-f319-47b1-981c-07754192b8db-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:04:08.234142 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:08.234113    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"
Jul 31 15:04:08.234288 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:08.234154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn" event={"ID":"ba36e8a4-f319-47b1-981c-07754192b8db","Type":"ContainerDied","Data":"8f1643c457603c50c1fff15fcf62c64faa606d05f3c86d747133009c63d2985f"}
Jul 31 15:04:08.234288 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:08.234198    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f1643c457603c50c1fff15fcf62c64faa606d05f3c86d747133009c63d2985f"
Jul 31 15:04:19.452654 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:19.452609    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"]
Jul 31 15:04:19.454772 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:19.454745    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-700618-gk8mn"]
Jul 31 15:04:19.579701 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:19.579663    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba36e8a4-f319-47b1-981c-07754192b8db" path="/var/lib/kubelet/pods/ba36e8a4-f319-47b1-981c-07754192b8db/volumes"
Jul 31 15:04:26.276762 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.276724    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"]
Jul 31 15:04:26.277425 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.277406    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba36e8a4-f319-47b1-981c-07754192b8db" containerName="revoke-keys"
Jul 31 15:04:26.277512 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.277428    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba36e8a4-f319-47b1-981c-07754192b8db" containerName="revoke-keys"
Jul 31 15:04:26.277570 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.277547    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba36e8a4-f319-47b1-981c-07754192b8db" containerName="revoke-keys"
Jul 31 15:04:26.279752 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.279732    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"
Jul 31 15:04:26.282431 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.282410    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:04:26.287960 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.287936    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"]
Jul 31 15:04:26.343533 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.343501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6\" (UID: \"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"
Jul 31 15:04:26.444304 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.444257    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6\" (UID: \"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"
Jul 31 15:04:26.444906 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.444885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6\" (UID: \"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"
Jul 31 15:04:26.593014 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.592983    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"
Jul 31 15:04:26.737691 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:26.737664    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"]
Jul 31 15:04:26.739220 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:04:26.739194    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e1ff3f_e06c_4bd1_abfc_903f5a405f8a.slice/crio-4beb00777c8035851c9110fae789cc3a4f6bc518dfacdb294e753106c24a17a1 WatchSource:0}: Error finding container 4beb00777c8035851c9110fae789cc3a4f6bc518dfacdb294e753106c24a17a1: Status 404 returned error can't find the container with id 4beb00777c8035851c9110fae789cc3a4f6bc518dfacdb294e753106c24a17a1
Jul 31 15:04:27.343286 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:27.343254    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6" event={"ID":"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a","Type":"ContainerStarted","Data":"90870e447c705c7cada21b88ac2fa305b0876dc9adbdaf8045cd44d5673defd2"}
Jul 31 15:04:27.343675 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:27.343307    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6" event={"ID":"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a","Type":"ContainerStarted","Data":"4beb00777c8035851c9110fae789cc3a4f6bc518dfacdb294e753106c24a17a1"}
Jul 31 15:04:27.360853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:27.360792    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6" podStartSLOduration=1.360774491 podStartE2EDuration="1.360774491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:04:27.357654182 +0000 UTC m=+3106.230217443" watchObservedRunningTime="2026-07-31 15:04:27.360774491 +0000 UTC m=+3106.233337745"
Jul 31 15:04:28.350290 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:28.350250    2558 generic.go:356] "Generic (PLEG): container finished" podID="d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a" containerID="90870e447c705c7cada21b88ac2fa305b0876dc9adbdaf8045cd44d5673defd2" exitCode=0
Jul 31 15:04:28.350681 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:28.350341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6" event={"ID":"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a","Type":"ContainerDied","Data":"90870e447c705c7cada21b88ac2fa305b0876dc9adbdaf8045cd44d5673defd2"}
Jul 31 15:04:29.490402 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:29.490379    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"
Jul 31 15:04:29.571545 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:29.571515    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a-maas-api-service-ca\") pod \"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a\" (UID: \"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a\") "
Jul 31 15:04:29.571918 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:29.571895    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a" (UID: "d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:29.672919 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:29.672834    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:04:30.361787 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:30.361750    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"
Jul 31 15:04:30.361787 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:30.361783    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6" event={"ID":"d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a","Type":"ContainerDied","Data":"4beb00777c8035851c9110fae789cc3a4f6bc518dfacdb294e753106c24a17a1"}
Jul 31 15:04:30.362014 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:30.361830    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4beb00777c8035851c9110fae789cc3a4f6bc518dfacdb294e753106c24a17a1"
Jul 31 15:04:41.647504 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:41.647471    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"]
Jul 31 15:04:41.648152 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:41.647743    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft" podUID="bf6be5b9-bf5f-45a0-9fca-f20659f274ba" containerName="maas-api" containerID="cri-o://330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf" gracePeriod=30
Jul 31 15:04:41.814327 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:41.814280    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"]
Jul 31 15:04:41.816425 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:41.816401    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2eeeef-5pjh6"]
Jul 31 15:04:41.905631 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:41.905570    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:04:42.093869 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.093835    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-openshift-service-ca\") pod \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") "
Jul 31 15:04:42.094092 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.093892    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8cjv\" (UniqueName: \"kubernetes.io/projected/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-kube-api-access-m8cjv\") pod \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") "
Jul 31 15:04:42.094092 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.093959    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls\") pod \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\" (UID: \"bf6be5b9-bf5f-45a0-9fca-f20659f274ba\") "
Jul 31 15:04:42.094310 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.094277    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf6be5b9-bf5f-45a0-9fca-f20659f274ba" (UID: "bf6be5b9-bf5f-45a0-9fca-f20659f274ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:04:42.096253 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.096221    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf6be5b9-bf5f-45a0-9fca-f20659f274ba" (UID: "bf6be5b9-bf5f-45a0-9fca-f20659f274ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:04:42.096364 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.096325    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-kube-api-access-m8cjv" (OuterVolumeSpecName: "kube-api-access-m8cjv") pod "bf6be5b9-bf5f-45a0-9fca-f20659f274ba" (UID: "bf6be5b9-bf5f-45a0-9fca-f20659f274ba"). InnerVolumeSpecName "kube-api-access-m8cjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:04:42.195457 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.195368    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-maas-api-tls\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.195457 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.195402    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-openshift-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.195457 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.195416    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8cjv\" (UniqueName: \"kubernetes.io/projected/bf6be5b9-bf5f-45a0-9fca-f20659f274ba-kube-api-access-m8cjv\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:04:42.429655 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.429619    2558 generic.go:356] "Generic (PLEG): container finished" podID="bf6be5b9-bf5f-45a0-9fca-f20659f274ba" containerID="330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf" exitCode=0
Jul 31 15:04:42.429838 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.429667    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"
Jul 31 15:04:42.429838 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.429681    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft" event={"ID":"bf6be5b9-bf5f-45a0-9fca-f20659f274ba","Type":"ContainerDied","Data":"330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf"}
Jul 31 15:04:42.429838 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.429739    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft" event={"ID":"bf6be5b9-bf5f-45a0-9fca-f20659f274ba","Type":"ContainerDied","Data":"f8928809da190241b42e18c16e0472648f1579f686e8d608cda32e4884e2861d"}
Jul 31 15:04:42.429838 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.429762    2558 scope.go:117] "RemoveContainer" containerID="330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf"
Jul 31 15:04:42.440792 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.440764    2558 scope.go:117] "RemoveContainer" containerID="330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf"
Jul 31 15:04:42.441083 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:04:42.441062    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf\": container with ID starting with 330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf not found: ID does not exist" containerID="330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf"
Jul 31 15:04:42.441167 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.441090    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf"} err="failed to get container status \"330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf\": rpc error: code = NotFound desc = could not find container \"330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf\": container with ID starting with 330a24863f6e0d3bd03a13c70c4d366faa48f6eb68cce894a4714adbf29200bf not found: ID does not exist"
Jul 31 15:04:42.454114 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.454043    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"]
Jul 31 15:04:42.457653 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:42.457617    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2eeeef-6f7845cd89-x25ft"]
Jul 31 15:04:43.579818 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:43.579784    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf6be5b9-bf5f-45a0-9fca-f20659f274ba" path="/var/lib/kubelet/pods/bf6be5b9-bf5f-45a0-9fca-f20659f274ba/volumes"
Jul 31 15:04:43.580212 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:43.580144    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a" path="/var/lib/kubelet/pods/d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a/volumes"
Jul 31 15:04:45.709355 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:04:45.709317    2558 scope.go:117] "RemoveContainer" containerID="ecf28d1ba631eae2afac04eb689061f6ae91f97648cda0f809b7b392535b8b4b"
Jul 31 15:05:22.371280 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.371242    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"]
Jul 31 15:05:22.371902 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.371887    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf6be5b9-bf5f-45a0-9fca-f20659f274ba" containerName="maas-api"
Jul 31 15:05:22.371953 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.371905    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf6be5b9-bf5f-45a0-9fca-f20659f274ba" containerName="maas-api"
Jul 31 15:05:22.371953 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.371919    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a" containerName="revoke-keys"
Jul 31 15:05:22.371953 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.371929    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a" containerName="revoke-keys"
Jul 31 15:05:22.372072 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.372016    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e1ff3f-e06c-4bd1-abfc-903f5a405f8a" containerName="revoke-keys"
Jul 31 15:05:22.372072 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.372041    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf6be5b9-bf5f-45a0-9fca-f20659f274ba" containerName="maas-api"
Jul 31 15:05:22.375651 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.375629    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.378151 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.378129    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-79a3ee-serving-cert\""
Jul 31 15:05:22.378549 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.378531    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cl2ws\""
Jul 31 15:05:22.383579 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.383558    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"]
Jul 31 15:05:22.468578 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.468538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lmth\" (UniqueName: \"kubernetes.io/projected/ba7f7208-206e-4f88-bcb3-024d48951024-kube-api-access-9lmth\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.468789 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.468651    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f7208-206e-4f88-bcb3-024d48951024-openshift-service-ca\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.468789 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.468755    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7f7208-206e-4f88-bcb3-024d48951024-maas-api-tls\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.569889 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.569851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmth\" (UniqueName: \"kubernetes.io/projected/ba7f7208-206e-4f88-bcb3-024d48951024-kube-api-access-9lmth\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.570095 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.569899    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f7208-206e-4f88-bcb3-024d48951024-openshift-service-ca\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.570095 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.569942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7f7208-206e-4f88-bcb3-024d48951024-maas-api-tls\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.570597 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.570573    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f7208-206e-4f88-bcb3-024d48951024-openshift-service-ca\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.572377 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.572354    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7f7208-206e-4f88-bcb3-024d48951024-maas-api-tls\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.578182 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.578161    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmth\" (UniqueName: \"kubernetes.io/projected/ba7f7208-206e-4f88-bcb3-024d48951024-kube-api-access-9lmth\") pod \"maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.690232 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.690148    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:22.841502 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:22.841472    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"]
Jul 31 15:05:22.844058 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:05:22.844002    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7f7208_206e_4f88_bcb3_024d48951024.slice/crio-ddbc735504d33f4112ed9935178caa3312dbd62eb2a4e7ba68b529f8f2844698 WatchSource:0}: Error finding container ddbc735504d33f4112ed9935178caa3312dbd62eb2a4e7ba68b529f8f2844698: Status 404 returned error can't find the container with id ddbc735504d33f4112ed9935178caa3312dbd62eb2a4e7ba68b529f8f2844698
Jul 31 15:05:23.656541 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:23.656503    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd" event={"ID":"ba7f7208-206e-4f88-bcb3-024d48951024","Type":"ContainerStarted","Data":"ddbc735504d33f4112ed9935178caa3312dbd62eb2a4e7ba68b529f8f2844698"}
Jul 31 15:05:27.680458 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:27.680424    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd" event={"ID":"ba7f7208-206e-4f88-bcb3-024d48951024","Type":"ContainerStarted","Data":"3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637"}
Jul 31 15:05:27.680882 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:27.680572    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:27.696971 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:27.696874    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd" podStartSLOduration=1.104876221 podStartE2EDuration="5.696857749s" totalImagesPullingTime="4.591981528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:05:27.695990622 +0000 UTC m=+3166.568553876" watchObservedRunningTime="2026-07-31 15:05:27.696857749 +0000 UTC m=+3166.569421002"
Jul 31 15:05:33.691432 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:33.691399    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:05:49.280292 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.280260    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"]
Jul 31 15:05:49.284080 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.284063    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.286677 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.286655    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"kube-root-ca.crt\""
Jul 31 15:05:49.288235 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.288219    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"auth-test-model-9966b2-kserve-self-signed-certs\""
Jul 31 15:05:49.288344 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.288231    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"default-dockercfg-lrnqb\""
Jul 31 15:05:49.288344 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.288235    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"openshift-service-ca.crt\""
Jul 31 15:05:49.292075 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.292054    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"]
Jul 31 15:05:49.310290 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.310267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tmp-dir\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.310403 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.310296    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np4ln\" (UniqueName: \"kubernetes.io/projected/18660dbe-8c9a-4b63-92d8-8aaf210322ae-kube-api-access-np4ln\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.310403 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.310385    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-dshm\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.310478 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.310452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-home\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.310478 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.310471    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tls-certs\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.310557 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.310491    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-model-cache\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.411704 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.411671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-dshm\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.411905 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.411729    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-home\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.411905 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.411756    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tls-certs\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.411905 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.411783    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-model-cache\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.411905 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.411857    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tmp-dir\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.411905 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.411879    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np4ln\" (UniqueName: \"kubernetes.io/projected/18660dbe-8c9a-4b63-92d8-8aaf210322ae-kube-api-access-np4ln\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.412207 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.412182    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-home\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.412272 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.412216    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-model-cache\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.412323 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.412284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tmp-dir\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.413923 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.413903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-dshm\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.414203 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.414187    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tls-certs\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.419801 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.419779    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-np4ln\" (UniqueName: \"kubernetes.io/projected/18660dbe-8c9a-4b63-92d8-8aaf210322ae-kube-api-access-np4ln\") pod \"auth-test-model-9966b2-kserve-757894759f-zqd88\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") " pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.594431 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.594399    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:49.734286 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.734253    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"]
Jul 31 15:05:49.803464 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:49.803436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88" event={"ID":"18660dbe-8c9a-4b63-92d8-8aaf210322ae","Type":"ContainerStarted","Data":"2ab929416c80068802eaf31ab667815dc7a4eeebdf05b8a97177747aa93fec07"}
Jul 31 15:05:50.810568 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:50.810527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88" event={"ID":"18660dbe-8c9a-4b63-92d8-8aaf210322ae","Type":"ContainerStarted","Data":"babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454"}
Jul 31 15:05:50.830083 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:50.829996    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88" podStartSLOduration=1.829975212 podStartE2EDuration="1.829975212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:05:50.82758808 +0000 UTC m=+3189.700151333" watchObservedRunningTime="2026-07-31 15:05:50.829975212 +0000 UTC m=+3189.702538470"
Jul 31 15:05:59.595110 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:59.595077    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:59.595586 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:59.595149    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:59.607342 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:59.607318    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:05:59.871308 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:05:59.871213    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:06:33.379183 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.379095    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"]
Jul 31 15:06:33.383509 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.383483    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.387505 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.387482    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"default-dockercfg-qpnpg\""
Jul 31 15:06:33.387695 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.387483    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"kube-root-ca.crt\""
Jul 31 15:06:33.387810 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.387491    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"openshift-service-ca.crt\""
Jul 31 15:06:33.387810 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.387550    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"auth-test-model-79a3ee-kserve-self-signed-certs\""
Jul 31 15:06:33.392663 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.392642    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"]
Jul 31 15:06:33.427509 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.427476    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr64l\" (UniqueName: \"kubernetes.io/projected/01d29421-dc87-46ac-adeb-72b97c9e557a-kube-api-access-mr64l\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.427664 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.427517    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-home\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.427664 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.427542    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-model-cache\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.427664 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.427614    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-dshm\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.427664 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.427649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d29421-dc87-46ac-adeb-72b97c9e557a-tls-certs\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.427815 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.427706    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-tmp-dir\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528285 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-tmp-dir\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528457 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr64l\" (UniqueName: \"kubernetes.io/projected/01d29421-dc87-46ac-adeb-72b97c9e557a-kube-api-access-mr64l\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528457 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-home\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528457 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528414    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-model-cache\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528632 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-dshm\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528632 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d29421-dc87-46ac-adeb-72b97c9e557a-tls-certs\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528744 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528710    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-tmp-dir\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528830 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528806    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-home\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.528882 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.528836    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-model-cache\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.530750 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.530728    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-dshm\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.530981 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.530965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d29421-dc87-46ac-adeb-72b97c9e557a-tls-certs\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.536429 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.536405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr64l\" (UniqueName: \"kubernetes.io/projected/01d29421-dc87-46ac-adeb-72b97c9e557a-kube-api-access-mr64l\") pod \"auth-test-model-79a3ee-kserve-6cf575699f-sh8ft\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") " pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.695396 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.695315    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:33.833297 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:33.833269    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"]
Jul 31 15:06:33.835223 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:06:33.835199    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01d29421_dc87_46ac_adeb_72b97c9e557a.slice/crio-68ccf42921b3ced214c4eea694f53ffb8e071e56b7c5ca0d6a8b5ba8bfc6bd37 WatchSource:0}: Error finding container 68ccf42921b3ced214c4eea694f53ffb8e071e56b7c5ca0d6a8b5ba8bfc6bd37: Status 404 returned error can't find the container with id 68ccf42921b3ced214c4eea694f53ffb8e071e56b7c5ca0d6a8b5ba8bfc6bd37
Jul 31 15:06:34.053262 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:34.053225    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft" event={"ID":"01d29421-dc87-46ac-adeb-72b97c9e557a","Type":"ContainerStarted","Data":"79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2"}
Jul 31 15:06:34.053439 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:34.053278    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft" event={"ID":"01d29421-dc87-46ac-adeb-72b97c9e557a","Type":"ContainerStarted","Data":"68ccf42921b3ced214c4eea694f53ffb8e071e56b7c5ca0d6a8b5ba8bfc6bd37"}
Jul 31 15:06:34.071829 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:34.071780    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft" podStartSLOduration=1.071763435 podStartE2EDuration="1.071763435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:06:34.069600219 +0000 UTC m=+3232.942163471" watchObservedRunningTime="2026-07-31 15:06:34.071763435 +0000 UTC m=+3232.944326731"
Jul 31 15:06:43.696045 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:43.695992    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:43.696045 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:43.696051    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:43.708366 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:43.708340    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:06:44.119786 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:06:44.119749    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:07:45.616659 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:07:45.616630    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:07:45.627848 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:07:45.627824    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:08:45.764230 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:08:45.764198    2558 scope.go:117] "RemoveContainer" containerID="b320c35e6049c2c048dde2246df1935372494cd1b9ae20d2834334e5cac49ccb"
Jul 31 15:09:45.179797 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:09:45.179711    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"]
Jul 31 15:09:45.785398 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:09:45.785365    2558 scope.go:117] "RemoveContainer" containerID="b5e2802b4890e5c67b1c1ba101da8abb37728727dfb8750745cba9c520121f0f"
Jul 31 15:09:46.183841 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:09:46.183799    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"]
Jul 31 15:10:00.199179 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.199129    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88" podUID="18660dbe-8c9a-4b63-92d8-8aaf210322ae" containerName="main" containerID="cri-o://babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454" gracePeriod=45
Jul 31 15:10:00.455343 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.455281    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:10:00.499616 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499589    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-model-cache\") pod \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") "
Jul 31 15:10:00.499802 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499644    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-home\") pod \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") "
Jul 31 15:10:00.499802 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499694    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tmp-dir\") pod \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") "
Jul 31 15:10:00.499802 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499711    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-np4ln\" (UniqueName: \"kubernetes.io/projected/18660dbe-8c9a-4b63-92d8-8aaf210322ae-kube-api-access-np4ln\") pod \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") "
Jul 31 15:10:00.499802 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499744    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-dshm\") pod \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") "
Jul 31 15:10:00.499802 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499768    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tls-certs\") pod \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\" (UID: \"18660dbe-8c9a-4b63-92d8-8aaf210322ae\") "
Jul 31 15:10:00.500118 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499878    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "18660dbe-8c9a-4b63-92d8-8aaf210322ae" (UID: "18660dbe-8c9a-4b63-92d8-8aaf210322ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:00.500118 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499982    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-home" (OuterVolumeSpecName: "home") pod "18660dbe-8c9a-4b63-92d8-8aaf210322ae" (UID: "18660dbe-8c9a-4b63-92d8-8aaf210322ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:00.500118 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.499997    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18660dbe-8c9a-4b63-92d8-8aaf210322ae" (UID: "18660dbe-8c9a-4b63-92d8-8aaf210322ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:00.500274 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.500215    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:00.500274 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.500234    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:00.500274 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.500249    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:00.501996 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.501969    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-dshm" (OuterVolumeSpecName: "dshm") pod "18660dbe-8c9a-4b63-92d8-8aaf210322ae" (UID: "18660dbe-8c9a-4b63-92d8-8aaf210322ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:00.502364 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.502338    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18660dbe-8c9a-4b63-92d8-8aaf210322ae" (UID: "18660dbe-8c9a-4b63-92d8-8aaf210322ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:10:00.502452 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.502399    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18660dbe-8c9a-4b63-92d8-8aaf210322ae-kube-api-access-np4ln" (OuterVolumeSpecName: "kube-api-access-np4ln") pod "18660dbe-8c9a-4b63-92d8-8aaf210322ae" (UID: "18660dbe-8c9a-4b63-92d8-8aaf210322ae"). InnerVolumeSpecName "kube-api-access-np4ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:10:00.601102 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.601061    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18660dbe-8c9a-4b63-92d8-8aaf210322ae-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:00.601102 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.601099    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np4ln\" (UniqueName: \"kubernetes.io/projected/18660dbe-8c9a-4b63-92d8-8aaf210322ae-kube-api-access-np4ln\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:00.601102 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:00.601109    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18660dbe-8c9a-4b63-92d8-8aaf210322ae-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:01.209144 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.209082    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft" podUID="01d29421-dc87-46ac-adeb-72b97c9e557a" containerName="main" containerID="cri-o://79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2" gracePeriod=45
Jul 31 15:10:01.222147 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.222115    2558 generic.go:356] "Generic (PLEG): container finished" podID="18660dbe-8c9a-4b63-92d8-8aaf210322ae" containerID="babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454" exitCode=0
Jul 31 15:10:01.222298 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.222169    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"
Jul 31 15:10:01.222298 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.222204    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88" event={"ID":"18660dbe-8c9a-4b63-92d8-8aaf210322ae","Type":"ContainerDied","Data":"babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454"}
Jul 31 15:10:01.222298 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.222257    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88" event={"ID":"18660dbe-8c9a-4b63-92d8-8aaf210322ae","Type":"ContainerDied","Data":"2ab929416c80068802eaf31ab667815dc7a4eeebdf05b8a97177747aa93fec07"}
Jul 31 15:10:01.222298 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.222278    2558 scope.go:117] "RemoveContainer" containerID="babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454"
Jul 31 15:10:01.295694 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.295664    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"]
Jul 31 15:10:01.297692 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.297671    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/auth-test-model-9966b2-kserve-757894759f-zqd88"]
Jul 31 15:10:01.321114 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.321085    2558 scope.go:117] "RemoveContainer" containerID="babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454"
Jul 31 15:10:01.321510 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:10:01.321425    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454\": container with ID starting with babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454 not found: ID does not exist" containerID="babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454"
Jul 31 15:10:01.321510 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.321460    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454"} err="failed to get container status \"babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454\": rpc error: code = NotFound desc = could not find container \"babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454\": container with ID starting with babcec46ebdf8698d7e0de3c4a5e1f45c976fab1e16181c02e4afeca6a976454 not found: ID does not exist"
Jul 31 15:10:01.465438 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.465379    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:10:01.510342 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510309    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-home\") pod \"01d29421-dc87-46ac-adeb-72b97c9e557a\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") "
Jul 31 15:10:01.510534 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510374    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-tmp-dir\") pod \"01d29421-dc87-46ac-adeb-72b97c9e557a\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") "
Jul 31 15:10:01.510534 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510456    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d29421-dc87-46ac-adeb-72b97c9e557a-tls-certs\") pod \"01d29421-dc87-46ac-adeb-72b97c9e557a\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") "
Jul 31 15:10:01.510534 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510535    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-model-cache\") pod \"01d29421-dc87-46ac-adeb-72b97c9e557a\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") "
Jul 31 15:10:01.510743 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510561    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-dshm\") pod \"01d29421-dc87-46ac-adeb-72b97c9e557a\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") "
Jul 31 15:10:01.510743 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510592    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr64l\" (UniqueName: \"kubernetes.io/projected/01d29421-dc87-46ac-adeb-72b97c9e557a-kube-api-access-mr64l\") pod \"01d29421-dc87-46ac-adeb-72b97c9e557a\" (UID: \"01d29421-dc87-46ac-adeb-72b97c9e557a\") "
Jul 31 15:10:01.510743 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510604    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01d29421-dc87-46ac-adeb-72b97c9e557a" (UID: "01d29421-dc87-46ac-adeb-72b97c9e557a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:01.510743 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510636    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-home" (OuterVolumeSpecName: "home") pod "01d29421-dc87-46ac-adeb-72b97c9e557a" (UID: "01d29421-dc87-46ac-adeb-72b97c9e557a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:01.510951 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510870    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:01.510951 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510891    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:01.510951 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.510866    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-model-cache" (OuterVolumeSpecName: "model-cache") pod "01d29421-dc87-46ac-adeb-72b97c9e557a" (UID: "01d29421-dc87-46ac-adeb-72b97c9e557a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:01.512772 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.512736    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d29421-dc87-46ac-adeb-72b97c9e557a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01d29421-dc87-46ac-adeb-72b97c9e557a" (UID: "01d29421-dc87-46ac-adeb-72b97c9e557a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:10:01.512772 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.512742    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-dshm" (OuterVolumeSpecName: "dshm") pod "01d29421-dc87-46ac-adeb-72b97c9e557a" (UID: "01d29421-dc87-46ac-adeb-72b97c9e557a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:01.512914 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.512881    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d29421-dc87-46ac-adeb-72b97c9e557a-kube-api-access-mr64l" (OuterVolumeSpecName: "kube-api-access-mr64l") pod "01d29421-dc87-46ac-adeb-72b97c9e557a" (UID: "01d29421-dc87-46ac-adeb-72b97c9e557a"). InnerVolumeSpecName "kube-api-access-mr64l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:10:01.580505 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.580470    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18660dbe-8c9a-4b63-92d8-8aaf210322ae" path="/var/lib/kubelet/pods/18660dbe-8c9a-4b63-92d8-8aaf210322ae/volumes"
Jul 31 15:10:01.612105 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.612015    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d29421-dc87-46ac-adeb-72b97c9e557a-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:01.612105 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.612068    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:01.612105 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.612078    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d29421-dc87-46ac-adeb-72b97c9e557a-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:01.612105 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:01.612086    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr64l\" (UniqueName: \"kubernetes.io/projected/01d29421-dc87-46ac-adeb-72b97c9e557a-kube-api-access-mr64l\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:10:02.229506 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.229471    2558 generic.go:356] "Generic (PLEG): container finished" podID="01d29421-dc87-46ac-adeb-72b97c9e557a" containerID="79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2" exitCode=0
Jul 31 15:10:02.229994 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.229525    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"
Jul 31 15:10:02.229994 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.229552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft" event={"ID":"01d29421-dc87-46ac-adeb-72b97c9e557a","Type":"ContainerDied","Data":"79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2"}
Jul 31 15:10:02.229994 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.229594    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft" event={"ID":"01d29421-dc87-46ac-adeb-72b97c9e557a","Type":"ContainerDied","Data":"68ccf42921b3ced214c4eea694f53ffb8e071e56b7c5ca0d6a8b5ba8bfc6bd37"}
Jul 31 15:10:02.229994 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.229613    2558 scope.go:117] "RemoveContainer" containerID="79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2"
Jul 31 15:10:02.238912 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.238890    2558 scope.go:117] "RemoveContainer" containerID="79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2"
Jul 31 15:10:02.239225 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:10:02.239200    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2\": container with ID starting with 79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2 not found: ID does not exist" containerID="79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2"
Jul 31 15:10:02.239279 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.239234    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2"} err="failed to get container status \"79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2\": rpc error: code = NotFound desc = could not find container \"79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2\": container with ID starting with 79105cad2810042d23a9de216c3d10dc4f2560d061ce1c0f1ada662673350fb2 not found: ID does not exist"
Jul 31 15:10:02.246694 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.246668    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"]
Jul 31 15:10:02.249633 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.249610    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/auth-test-model-79a3ee-kserve-6cf575699f-sh8ft"]
Jul 31 15:10:02.880289 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.880255    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"]
Jul 31 15:10:02.880678 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.880666    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d29421-dc87-46ac-adeb-72b97c9e557a" containerName="main"
Jul 31 15:10:02.880738 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.880679    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d29421-dc87-46ac-adeb-72b97c9e557a" containerName="main"
Jul 31 15:10:02.880738 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.880697    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18660dbe-8c9a-4b63-92d8-8aaf210322ae" containerName="main"
Jul 31 15:10:02.880738 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.880704    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="18660dbe-8c9a-4b63-92d8-8aaf210322ae" containerName="main"
Jul 31 15:10:02.880846 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.880780    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d29421-dc87-46ac-adeb-72b97c9e557a" containerName="main"
Jul 31 15:10:02.880846 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.880787    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="18660dbe-8c9a-4b63-92d8-8aaf210322ae" containerName="main"
Jul 31 15:10:02.884985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.884969    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:02.889126 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.889102    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"default-dockercfg-lrnqb\""
Jul 31 15:10:02.889240 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.889140    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"kube-root-ca.crt\""
Jul 31 15:10:02.889240 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.889159    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"openshift-service-ca.crt\""
Jul 31 15:10:02.889240 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.889140    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9966b2\"/\"sub-test-model-9966b2-kserve-self-signed-certs\""
Jul 31 15:10:02.894116 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.894096    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"]
Jul 31 15:10:02.922307 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.922276    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-model-cache\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:02.922445 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.922334    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cfb89c-3597-49aa-8aa0-72386b806f87-tls-certs\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:02.922445 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.922358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-dshm\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:02.922445 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.922416    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-home\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:02.922445 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.922439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44jn4\" (UniqueName: \"kubernetes.io/projected/a5cfb89c-3597-49aa-8aa0-72386b806f87-kube-api-access-44jn4\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:02.922603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:02.922505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-tmp-dir\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024010 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.023975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-dshm\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024222 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.024041    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-home\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024222 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.024069    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44jn4\" (UniqueName: \"kubernetes.io/projected/a5cfb89c-3597-49aa-8aa0-72386b806f87-kube-api-access-44jn4\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024222 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.024101    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-tmp-dir\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024222 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.024213    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-model-cache\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024449 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.024303    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cfb89c-3597-49aa-8aa0-72386b806f87-tls-certs\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024593 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.024563    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-model-cache\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024593 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.024579    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-tmp-dir\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.024788 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.024768    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-home\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.026379 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.026352    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-dshm\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.026722 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.026706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cfb89c-3597-49aa-8aa0-72386b806f87-tls-certs\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.031977 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.031953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44jn4\" (UniqueName: \"kubernetes.io/projected/a5cfb89c-3597-49aa-8aa0-72386b806f87-kube-api-access-44jn4\") pod \"sub-test-model-9966b2-kserve-555bc86f9f-zspw8\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") " pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.196643 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.196541    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:03.331198 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.331170    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"]
Jul 31 15:10:03.333453 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:10:03.333420    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5cfb89c_3597_49aa_8aa0_72386b806f87.slice/crio-335d243fe091bd6651a578fb67476cfa5e3d95ce86ce0e7aac7e4baa73880748 WatchSource:0}: Error finding container 335d243fe091bd6651a578fb67476cfa5e3d95ce86ce0e7aac7e4baa73880748: Status 404 returned error can't find the container with id 335d243fe091bd6651a578fb67476cfa5e3d95ce86ce0e7aac7e4baa73880748
Jul 31 15:10:03.580868 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:03.580833    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d29421-dc87-46ac-adeb-72b97c9e557a" path="/var/lib/kubelet/pods/01d29421-dc87-46ac-adeb-72b97c9e557a/volumes"
Jul 31 15:10:04.245857 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:04.245824    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8" event={"ID":"a5cfb89c-3597-49aa-8aa0-72386b806f87","Type":"ContainerStarted","Data":"91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2"}
Jul 31 15:10:04.246060 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:04.245867    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8" event={"ID":"a5cfb89c-3597-49aa-8aa0-72386b806f87","Type":"ContainerStarted","Data":"335d243fe091bd6651a578fb67476cfa5e3d95ce86ce0e7aac7e4baa73880748"}
Jul 31 15:10:04.263320 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:04.263274    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8" podStartSLOduration=2.263259503 podStartE2EDuration="2.263259503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:04.26134669 +0000 UTC m=+3443.133909943" watchObservedRunningTime="2026-07-31 15:10:04.263259503 +0000 UTC m=+3443.135822756"
Jul 31 15:10:13.196797 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:13.196751    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:13.197229 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:13.196812    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:13.209538 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:13.209509    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:13.303579 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:13.303552    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:10:24.784184 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.783831    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"]
Jul 31 15:10:24.789962 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.789933    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:24.794628 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.794590    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"sub-test-model-79a3ee-kserve-self-signed-certs\""
Jul 31 15:10:24.794947 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.794913    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"kube-root-ca.crt\""
Jul 31 15:10:24.795208 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.795014    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"openshift-service-ca.crt\""
Jul 31 15:10:24.795339 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.795273    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"default-dockercfg-qpnpg\""
Jul 31 15:10:24.797810 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.797789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"]
Jul 31 15:10:24.934433 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.934395    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-model-cache\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:24.934433 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.934436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-home\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:24.934691 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.934463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d0826f-9b46-415b-a626-a39415ae71b0-tls-certs\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:24.934691 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.934545    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-dshm\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:24.934691 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.934602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-tmp-dir\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:24.934691 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:24.934653    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7hbr\" (UniqueName: \"kubernetes.io/projected/d9d0826f-9b46-415b-a626-a39415ae71b0-kube-api-access-k7hbr\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.035430 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035340    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-tmp-dir\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.035430 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7hbr\" (UniqueName: \"kubernetes.io/projected/d9d0826f-9b46-415b-a626-a39415ae71b0-kube-api-access-k7hbr\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.035694 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-model-cache\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.035694 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-home\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.035694 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d0826f-9b46-415b-a626-a39415ae71b0-tls-certs\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.035694 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-dshm\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.035922 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035879    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-tmp-dir\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.035983 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035962    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-model-cache\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.036060 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.035963    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-home\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.037999 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.037972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-dshm\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.038276 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.038261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d0826f-9b46-415b-a626-a39415ae71b0-tls-certs\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.042988 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.042969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7hbr\" (UniqueName: \"kubernetes.io/projected/d9d0826f-9b46-415b-a626-a39415ae71b0-kube-api-access-k7hbr\") pod \"sub-test-model-79a3ee-kserve-d45bbd75f-2fndn\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") " pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.103017 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.102976    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:25.243461 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.243432    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"]
Jul 31 15:10:25.244485 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:10:25.244460    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9d0826f_9b46_415b_a626_a39415ae71b0.slice/crio-1dbe0e6225f2721e93930fdae3ba7d9e422e683d5c96e7468b4f9ba1de385706 WatchSource:0}: Error finding container 1dbe0e6225f2721e93930fdae3ba7d9e422e683d5c96e7468b4f9ba1de385706: Status 404 returned error can't find the container with id 1dbe0e6225f2721e93930fdae3ba7d9e422e683d5c96e7468b4f9ba1de385706
Jul 31 15:10:25.363013 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.362413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn" event={"ID":"d9d0826f-9b46-415b-a626-a39415ae71b0","Type":"ContainerStarted","Data":"6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7"}
Jul 31 15:10:25.363013 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.362466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn" event={"ID":"d9d0826f-9b46-415b-a626-a39415ae71b0","Type":"ContainerStarted","Data":"1dbe0e6225f2721e93930fdae3ba7d9e422e683d5c96e7468b4f9ba1de385706"}
Jul 31 15:10:25.380249 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:25.380203    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn" podStartSLOduration=1.3801862520000001 podStartE2EDuration="1.380186252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:25.37788093 +0000 UTC m=+3464.250444184" watchObservedRunningTime="2026-07-31 15:10:25.380186252 +0000 UTC m=+3464.252749540"
Jul 31 15:10:35.103384 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:35.103341    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:35.103799 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:35.103404    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:35.116046 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:35.115997    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:35.429547 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:35.429456    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:10:45.806399 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:45.806368    2558 scope.go:117] "RemoveContainer" containerID="90870e447c705c7cada21b88ac2fa305b0876dc9adbdaf8045cd44d5673defd2"
Jul 31 15:10:45.815104 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:10:45.815082    2558 scope.go:117] "RemoveContainer" containerID="6f04ad6ebe4be4a820e0a6b1d9562b7ba91b37346cab8f0f557f3ea784a7a4e6"
Jul 31 15:11:37.318011 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:37.317972    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"]
Jul 31 15:11:38.254806 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:38.254769    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"]
Jul 31 15:11:52.340052 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.339973    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8" podUID="a5cfb89c-3597-49aa-8aa0-72386b806f87" containerName="main" containerID="cri-o://91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2" gracePeriod=45
Jul 31 15:11:52.607127 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.607048    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:11:52.688035 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688002    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44jn4\" (UniqueName: \"kubernetes.io/projected/a5cfb89c-3597-49aa-8aa0-72386b806f87-kube-api-access-44jn4\") pod \"a5cfb89c-3597-49aa-8aa0-72386b806f87\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") "
Jul 31 15:11:52.688228 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688104    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-home\") pod \"a5cfb89c-3597-49aa-8aa0-72386b806f87\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") "
Jul 31 15:11:52.688228 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688204    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-tmp-dir\") pod \"a5cfb89c-3597-49aa-8aa0-72386b806f87\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") "
Jul 31 15:11:52.688356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688253    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-model-cache\") pod \"a5cfb89c-3597-49aa-8aa0-72386b806f87\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") "
Jul 31 15:11:52.688356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688292    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cfb89c-3597-49aa-8aa0-72386b806f87-tls-certs\") pod \"a5cfb89c-3597-49aa-8aa0-72386b806f87\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") "
Jul 31 15:11:52.688356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688325    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-dshm\") pod \"a5cfb89c-3597-49aa-8aa0-72386b806f87\" (UID: \"a5cfb89c-3597-49aa-8aa0-72386b806f87\") "
Jul 31 15:11:52.688510 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688382    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-home" (OuterVolumeSpecName: "home") pod "a5cfb89c-3597-49aa-8aa0-72386b806f87" (UID: "a5cfb89c-3597-49aa-8aa0-72386b806f87"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:52.688510 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688420    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5cfb89c-3597-49aa-8aa0-72386b806f87" (UID: "a5cfb89c-3597-49aa-8aa0-72386b806f87"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:52.688510 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688486    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5cfb89c-3597-49aa-8aa0-72386b806f87" (UID: "a5cfb89c-3597-49aa-8aa0-72386b806f87"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:52.688704 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688687    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:52.688756 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688705    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:52.688756 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.688716    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:52.690252 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.690226    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cfb89c-3597-49aa-8aa0-72386b806f87-kube-api-access-44jn4" (OuterVolumeSpecName: "kube-api-access-44jn4") pod "a5cfb89c-3597-49aa-8aa0-72386b806f87" (UID: "a5cfb89c-3597-49aa-8aa0-72386b806f87"). InnerVolumeSpecName "kube-api-access-44jn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:52.690465 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.690441    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cfb89c-3597-49aa-8aa0-72386b806f87-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5cfb89c-3597-49aa-8aa0-72386b806f87" (UID: "a5cfb89c-3597-49aa-8aa0-72386b806f87"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:52.690522 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.690479    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-dshm" (OuterVolumeSpecName: "dshm") pod "a5cfb89c-3597-49aa-8aa0-72386b806f87" (UID: "a5cfb89c-3597-49aa-8aa0-72386b806f87"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:52.789985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.789947    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5cfb89c-3597-49aa-8aa0-72386b806f87-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:52.789985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.789982    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44jn4\" (UniqueName: \"kubernetes.io/projected/a5cfb89c-3597-49aa-8aa0-72386b806f87-kube-api-access-44jn4\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:52.790233 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.789996    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5cfb89c-3597-49aa-8aa0-72386b806f87-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:52.838873 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.838838    2558 generic.go:356] "Generic (PLEG): container finished" podID="a5cfb89c-3597-49aa-8aa0-72386b806f87" containerID="91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2" exitCode=0
Jul 31 15:11:52.839076 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.838883    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"
Jul 31 15:11:52.839076 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.838930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8" event={"ID":"a5cfb89c-3597-49aa-8aa0-72386b806f87","Type":"ContainerDied","Data":"91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2"}
Jul 31 15:11:52.839076 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.838983    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8" event={"ID":"a5cfb89c-3597-49aa-8aa0-72386b806f87","Type":"ContainerDied","Data":"335d243fe091bd6651a578fb67476cfa5e3d95ce86ce0e7aac7e4baa73880748"}
Jul 31 15:11:52.839076 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.839006    2558 scope.go:117] "RemoveContainer" containerID="91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2"
Jul 31 15:11:52.848457 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.848434    2558 scope.go:117] "RemoveContainer" containerID="91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2"
Jul 31 15:11:52.848696 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:11:52.848680    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2\": container with ID starting with 91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2 not found: ID does not exist" containerID="91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2"
Jul 31 15:11:52.848748 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.848703    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2"} err="failed to get container status \"91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2\": rpc error: code = NotFound desc = could not find container \"91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2\": container with ID starting with 91aa0d173dff61643048b9364c2ba60d79e5b8055ad2f6af987697f53142fba2 not found: ID does not exist"
Jul 31 15:11:52.862706 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.862643    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"]
Jul 31 15:11:52.864696 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:52.864678    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9966b2/sub-test-model-9966b2-kserve-555bc86f9f-zspw8"]
Jul 31 15:11:53.274679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.274582    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn" podUID="d9d0826f-9b46-415b-a626-a39415ae71b0" containerName="main" containerID="cri-o://6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7" gracePeriod=45
Jul 31 15:11:53.540598 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.540573    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:11:53.579542 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.579507    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5cfb89c-3597-49aa-8aa0-72386b806f87" path="/var/lib/kubelet/pods/a5cfb89c-3597-49aa-8aa0-72386b806f87/volumes"
Jul 31 15:11:53.597642 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.597618    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d0826f-9b46-415b-a626-a39415ae71b0-tls-certs\") pod \"d9d0826f-9b46-415b-a626-a39415ae71b0\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") "
Jul 31 15:11:53.597779 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.597724    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-tmp-dir\") pod \"d9d0826f-9b46-415b-a626-a39415ae71b0\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") "
Jul 31 15:11:53.597779 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.597746    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7hbr\" (UniqueName: \"kubernetes.io/projected/d9d0826f-9b46-415b-a626-a39415ae71b0-kube-api-access-k7hbr\") pod \"d9d0826f-9b46-415b-a626-a39415ae71b0\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") "
Jul 31 15:11:53.597890 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.597785    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-model-cache\") pod \"d9d0826f-9b46-415b-a626-a39415ae71b0\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") "
Jul 31 15:11:53.597890 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.597823    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-dshm\") pod \"d9d0826f-9b46-415b-a626-a39415ae71b0\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") "
Jul 31 15:11:53.597890 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.597876    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-home\") pod \"d9d0826f-9b46-415b-a626-a39415ae71b0\" (UID: \"d9d0826f-9b46-415b-a626-a39415ae71b0\") "
Jul 31 15:11:53.598317 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.597984    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9d0826f-9b46-415b-a626-a39415ae71b0" (UID: "d9d0826f-9b46-415b-a626-a39415ae71b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:53.598317 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.598139    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9d0826f-9b46-415b-a626-a39415ae71b0" (UID: "d9d0826f-9b46-415b-a626-a39415ae71b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:53.598317 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.598200    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-home" (OuterVolumeSpecName: "home") pod "d9d0826f-9b46-415b-a626-a39415ae71b0" (UID: "d9d0826f-9b46-415b-a626-a39415ae71b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:53.598317 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.598296    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:53.599943 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.599899    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d0826f-9b46-415b-a626-a39415ae71b0-kube-api-access-k7hbr" (OuterVolumeSpecName: "kube-api-access-k7hbr") pod "d9d0826f-9b46-415b-a626-a39415ae71b0" (UID: "d9d0826f-9b46-415b-a626-a39415ae71b0"). InnerVolumeSpecName "kube-api-access-k7hbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:53.600088 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.600054    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-dshm" (OuterVolumeSpecName: "dshm") pod "d9d0826f-9b46-415b-a626-a39415ae71b0" (UID: "d9d0826f-9b46-415b-a626-a39415ae71b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:53.600257 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.600238    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d0826f-9b46-415b-a626-a39415ae71b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9d0826f-9b46-415b-a626-a39415ae71b0" (UID: "d9d0826f-9b46-415b-a626-a39415ae71b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:53.699697 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.699657    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7hbr\" (UniqueName: \"kubernetes.io/projected/d9d0826f-9b46-415b-a626-a39415ae71b0-kube-api-access-k7hbr\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:53.699697 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.699692    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:53.699697 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.699702    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:53.699947 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.699710    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d0826f-9b46-415b-a626-a39415ae71b0-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:53.699947 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.699718    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d0826f-9b46-415b-a626-a39415ae71b0-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:11:53.846738 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.846644    2558 generic.go:356] "Generic (PLEG): container finished" podID="d9d0826f-9b46-415b-a626-a39415ae71b0" containerID="6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7" exitCode=0
Jul 31 15:11:53.846738 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.846693    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"
Jul 31 15:11:53.846955 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.846735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn" event={"ID":"d9d0826f-9b46-415b-a626-a39415ae71b0","Type":"ContainerDied","Data":"6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7"}
Jul 31 15:11:53.846955 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.846776    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn" event={"ID":"d9d0826f-9b46-415b-a626-a39415ae71b0","Type":"ContainerDied","Data":"1dbe0e6225f2721e93930fdae3ba7d9e422e683d5c96e7468b4f9ba1de385706"}
Jul 31 15:11:53.846955 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.846796    2558 scope.go:117] "RemoveContainer" containerID="6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7"
Jul 31 15:11:53.856018 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.856001    2558 scope.go:117] "RemoveContainer" containerID="6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7"
Jul 31 15:11:53.856276 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:11:53.856251    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7\": container with ID starting with 6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7 not found: ID does not exist" containerID="6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7"
Jul 31 15:11:53.856328 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.856311    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7"} err="failed to get container status \"6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7\": rpc error: code = NotFound desc = could not find container \"6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7\": container with ID starting with 6f787f182a239a3f153a8ae06f807191a8cf98e7e73293cc3d7993ea8edcebb7 not found: ID does not exist"
Jul 31 15:11:53.868356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.868332    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"]
Jul 31 15:11:53.873789 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:53.873767    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/sub-test-model-79a3ee-kserve-d45bbd75f-2fndn"]
Jul 31 15:11:55.580715 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:11:55.580686    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d0826f-9b46-415b-a626-a39415ae71b0" path="/var/lib/kubelet/pods/d9d0826f-9b46-415b-a626-a39415ae71b0/volumes"
Jul 31 15:12:22.784758 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.784724    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"]
Jul 31 15:12:22.785374 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.785354    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5cfb89c-3597-49aa-8aa0-72386b806f87" containerName="main"
Jul 31 15:12:22.785461 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.785377    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5cfb89c-3597-49aa-8aa0-72386b806f87" containerName="main"
Jul 31 15:12:22.785461 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.785397    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d0826f-9b46-415b-a626-a39415ae71b0" containerName="main"
Jul 31 15:12:22.785461 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.785405    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d0826f-9b46-415b-a626-a39415ae71b0" containerName="main"
Jul 31 15:12:22.785612 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.785507    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d0826f-9b46-415b-a626-a39415ae71b0" containerName="main"
Jul 31 15:12:22.785612 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.785528    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5cfb89c-3597-49aa-8aa0-72386b806f87" containerName="main"
Jul 31 15:12:22.788843 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.788824    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:22.791888 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.791861    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"openshift-service-ca.crt\""
Jul 31 15:12:22.793050 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.793013    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"kube-root-ca.crt\""
Jul 31 15:12:22.793175 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.793108    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"rate-test-model-79a3ee-kserve-self-signed-certs\""
Jul 31 15:12:22.793243 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.793167    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-79a3ee\"/\"default-dockercfg-qpnpg\""
Jul 31 15:12:22.798283 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.798263    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"]
Jul 31 15:12:22.980666 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.980630    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-home\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:22.980851 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.980679    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9h4x\" (UniqueName: \"kubernetes.io/projected/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-kube-api-access-h9h4x\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:22.980851 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.980716    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-model-cache\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:22.980851 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.980783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tmp-dir\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:22.980960 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.980854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-dshm\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:22.980960 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:22.980899    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tls-certs\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.081508 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.081473    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-dshm\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.081761 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.081669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tls-certs\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.081914 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.081868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-home\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.082073 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.082040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9h4x\" (UniqueName: \"kubernetes.io/projected/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-kube-api-access-h9h4x\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.082281 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.082264    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-model-cache\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.082423 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.082409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tmp-dir\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.082681 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.082653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-model-cache\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.082909 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.082889    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tmp-dir\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.083061 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.083042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-home\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.084153 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.084132    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-dshm\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.084604 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.084585    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tls-certs\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.093724 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.093699    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9h4x\" (UniqueName: \"kubernetes.io/projected/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-kube-api-access-h9h4x\") pod \"rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") " pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.099740 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.099716    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:23.243806 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:23.243775    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"]
Jul 31 15:12:23.245497 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:12:23.245471    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee39e5a3_11be_4dde_ac8a_30d3cf6e070b.slice/crio-c5a2f3fa5b42e37c1aae3ef52141604567d0f9369edf8fbf012964ddda104650 WatchSource:0}: Error finding container c5a2f3fa5b42e37c1aae3ef52141604567d0f9369edf8fbf012964ddda104650: Status 404 returned error can't find the container with id c5a2f3fa5b42e37c1aae3ef52141604567d0f9369edf8fbf012964ddda104650
Jul 31 15:12:24.011545 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:24.011511    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r" event={"ID":"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b","Type":"ContainerStarted","Data":"00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571"}
Jul 31 15:12:24.012168 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:24.011560    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r" event={"ID":"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b","Type":"ContainerStarted","Data":"c5a2f3fa5b42e37c1aae3ef52141604567d0f9369edf8fbf012964ddda104650"}
Jul 31 15:12:24.032676 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:24.032624    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r" podStartSLOduration=2.032608496 podStartE2EDuration="2.032608496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:12:24.030085895 +0000 UTC m=+3582.902649146" watchObservedRunningTime="2026-07-31 15:12:24.032608496 +0000 UTC m=+3582.905171748"
Jul 31 15:12:33.100139 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:33.100100    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:33.100139 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:33.100139    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:33.112661 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:33.112638    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:34.079131 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:34.079101    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:12:45.653819 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:45.653788    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:12:45.661648 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:12:45.661625    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:13:37.192276 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:37.192240    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"]
Jul 31 15:13:38.230612 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.230571    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7480-140695933443504"]
Jul 31 15:13:38.234230 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.234213    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:38.237371 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.237344    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-m5t4f\""
Jul 31 15:13:38.238229 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.238200    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933443504"]
Jul 31 15:13:38.333373 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.333336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rbp\" (UniqueName: \"kubernetes.io/projected/e2786588-0104-4cd4-8507-0983c823d562-kube-api-access-g2rbp\") pod \"test-curl-7480-140695933443504\" (UID: \"e2786588-0104-4cd4-8507-0983c823d562\") " pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:38.434610 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.434571    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rbp\" (UniqueName: \"kubernetes.io/projected/e2786588-0104-4cd4-8507-0983c823d562-kube-api-access-g2rbp\") pod \"test-curl-7480-140695933443504\" (UID: \"e2786588-0104-4cd4-8507-0983c823d562\") " pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:38.442267 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.442238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rbp\" (UniqueName: \"kubernetes.io/projected/e2786588-0104-4cd4-8507-0983c823d562-kube-api-access-g2rbp\") pod \"test-curl-7480-140695933443504\" (UID: \"e2786588-0104-4cd4-8507-0983c823d562\") " pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:38.545455 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.545430    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:38.676875 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.676843    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933443504"]
Jul 31 15:13:38.680703 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:13:38.680675    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2786588_0104_4cd4_8507_0983c823d562.slice/crio-7aae8a9997e817188d74f8afd928d5dc2d9237ebbfca87b5e44d57520bb5757e WatchSource:0}: Error finding container 7aae8a9997e817188d74f8afd928d5dc2d9237ebbfca87b5e44d57520bb5757e: Status 404 returned error can't find the container with id 7aae8a9997e817188d74f8afd928d5dc2d9237ebbfca87b5e44d57520bb5757e
Jul 31 15:13:38.682401 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:38.682382    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:13:39.435972 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:39.435940    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933443504" event={"ID":"e2786588-0104-4cd4-8507-0983c823d562","Type":"ContainerStarted","Data":"7aae8a9997e817188d74f8afd928d5dc2d9237ebbfca87b5e44d57520bb5757e"}
Jul 31 15:13:40.442662 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:40.442621    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2786588-0104-4cd4-8507-0983c823d562" containerID="0cf7c25386916890f62405bfdec10be98a11ae387eed124e80dbacccd7ce3c87" exitCode=0
Jul 31 15:13:40.443087 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:40.442704    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933443504" event={"ID":"e2786588-0104-4cd4-8507-0983c823d562","Type":"ContainerDied","Data":"0cf7c25386916890f62405bfdec10be98a11ae387eed124e80dbacccd7ce3c87"}
Jul 31 15:13:41.589309 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.589282    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:41.621571 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.621540    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7480-140695933443504_e2786588-0104-4cd4-8507-0983c823d562/test-curl-7480-140695933443504/0.log"
Jul 31 15:13:41.645669 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.645641    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7480-140695933443504"]
Jul 31 15:13:41.647658 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.647625    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7480-140695933443504"]
Jul 31 15:13:41.665182 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.665158    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2rbp\" (UniqueName: \"kubernetes.io/projected/e2786588-0104-4cd4-8507-0983c823d562-kube-api-access-g2rbp\") pod \"e2786588-0104-4cd4-8507-0983c823d562\" (UID: \"e2786588-0104-4cd4-8507-0983c823d562\") "
Jul 31 15:13:41.667321 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.667299    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2786588-0104-4cd4-8507-0983c823d562-kube-api-access-g2rbp" (OuterVolumeSpecName: "kube-api-access-g2rbp") pod "e2786588-0104-4cd4-8507-0983c823d562" (UID: "e2786588-0104-4cd4-8507-0983c823d562"). InnerVolumeSpecName "kube-api-access-g2rbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:41.766365 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.766290    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rbp\" (UniqueName: \"kubernetes.io/projected/e2786588-0104-4cd4-8507-0983c823d562-kube-api-access-g2rbp\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:41.783223 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.783194    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7480-140695933441328"]
Jul 31 15:13:41.783618 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.783605    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2786588-0104-4cd4-8507-0983c823d562" containerName="test-curl-7480-140695933443504"
Jul 31 15:13:41.783670 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.783619    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2786588-0104-4cd4-8507-0983c823d562" containerName="test-curl-7480-140695933443504"
Jul 31 15:13:41.783722 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.783712    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2786588-0104-4cd4-8507-0983c823d562" containerName="test-curl-7480-140695933443504"
Jul 31 15:13:41.786791 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.786777    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933441328"
Jul 31 15:13:41.793524 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.793499    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933441328"]
Jul 31 15:13:41.867125 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.867094    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmrh\" (UniqueName: \"kubernetes.io/projected/808cab79-ef01-46ad-a3a7-1bb36567c4c2-kube-api-access-vbmrh\") pod \"test-curl-7480-140695933441328\" (UID: \"808cab79-ef01-46ad-a3a7-1bb36567c4c2\") " pod="opendatahub/test-curl-7480-140695933441328"
Jul 31 15:13:41.968076 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.968018    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmrh\" (UniqueName: \"kubernetes.io/projected/808cab79-ef01-46ad-a3a7-1bb36567c4c2-kube-api-access-vbmrh\") pod \"test-curl-7480-140695933441328\" (UID: \"808cab79-ef01-46ad-a3a7-1bb36567c4c2\") " pod="opendatahub/test-curl-7480-140695933441328"
Jul 31 15:13:41.975720 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:41.975689    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmrh\" (UniqueName: \"kubernetes.io/projected/808cab79-ef01-46ad-a3a7-1bb36567c4c2-kube-api-access-vbmrh\") pod \"test-curl-7480-140695933441328\" (UID: \"808cab79-ef01-46ad-a3a7-1bb36567c4c2\") " pod="opendatahub/test-curl-7480-140695933441328"
Jul 31 15:13:42.097504 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:42.097469    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933441328"
Jul 31 15:13:42.245078 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:42.245053    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933441328"]
Jul 31 15:13:42.247487 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:13:42.247462    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod808cab79_ef01_46ad_a3a7_1bb36567c4c2.slice/crio-473b9b36d82e3efbefd19baf31d7180d2721a887ecb2a3b13e464f5241045779 WatchSource:0}: Error finding container 473b9b36d82e3efbefd19baf31d7180d2721a887ecb2a3b13e464f5241045779: Status 404 returned error can't find the container with id 473b9b36d82e3efbefd19baf31d7180d2721a887ecb2a3b13e464f5241045779
Jul 31 15:13:42.456532 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:42.456447    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:42.456532 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:42.456474    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aae8a9997e817188d74f8afd928d5dc2d9237ebbfca87b5e44d57520bb5757e"
Jul 31 15:13:42.457606 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:42.457578    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933441328" event={"ID":"808cab79-ef01-46ad-a3a7-1bb36567c4c2","Type":"ContainerStarted","Data":"473b9b36d82e3efbefd19baf31d7180d2721a887ecb2a3b13e464f5241045779"}
Jul 31 15:13:42.467851 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:13:42.467825    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7480-140695933443504\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="e2786588-0104-4cd4-8507-0983c823d562" pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:43.464660 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:43.464626    2558 generic.go:356] "Generic (PLEG): container finished" podID="808cab79-ef01-46ad-a3a7-1bb36567c4c2" containerID="e6fc74928be2de3a2d1ee8fd94933a1c9e73ca2b9ff5df84f291c7337482a13f" exitCode=0
Jul 31 15:13:43.465147 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:43.464708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933441328" event={"ID":"808cab79-ef01-46ad-a3a7-1bb36567c4c2","Type":"ContainerDied","Data":"e6fc74928be2de3a2d1ee8fd94933a1c9e73ca2b9ff5df84f291c7337482a13f"}
Jul 31 15:13:43.467065 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:13:43.467038    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7480-140695933443504\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="e2786588-0104-4cd4-8507-0983c823d562" pod="opendatahub/test-curl-7480-140695933443504"
Jul 31 15:13:43.580731 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:43.580700    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2786588-0104-4cd4-8507-0983c823d562" path="/var/lib/kubelet/pods/e2786588-0104-4cd4-8507-0983c823d562/volumes"
Jul 31 15:13:44.615865 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.615838    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933441328"
Jul 31 15:13:44.637920 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.637895    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7480-140695933441328_808cab79-ef01-46ad-a3a7-1bb36567c4c2/test-curl-7480-140695933441328/0.log"
Jul 31 15:13:44.662790 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.662762    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7480-140695933441328"]
Jul 31 15:13:44.666348 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.666323    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7480-140695933441328"]
Jul 31 15:13:44.695314 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.695287    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbmrh\" (UniqueName: \"kubernetes.io/projected/808cab79-ef01-46ad-a3a7-1bb36567c4c2-kube-api-access-vbmrh\") pod \"808cab79-ef01-46ad-a3a7-1bb36567c4c2\" (UID: \"808cab79-ef01-46ad-a3a7-1bb36567c4c2\") "
Jul 31 15:13:44.697483 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.697459    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/808cab79-ef01-46ad-a3a7-1bb36567c4c2-kube-api-access-vbmrh" (OuterVolumeSpecName: "kube-api-access-vbmrh") pod "808cab79-ef01-46ad-a3a7-1bb36567c4c2" (UID: "808cab79-ef01-46ad-a3a7-1bb36567c4c2"). InnerVolumeSpecName "kube-api-access-vbmrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:44.796613 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.796584    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbmrh\" (UniqueName: \"kubernetes.io/projected/808cab79-ef01-46ad-a3a7-1bb36567c4c2-kube-api-access-vbmrh\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:44.844400 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.844367    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:44.844833 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.844818    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="808cab79-ef01-46ad-a3a7-1bb36567c4c2" containerName="test-curl-7480-140695933441328"
Jul 31 15:13:44.844882 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.844835    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="808cab79-ef01-46ad-a3a7-1bb36567c4c2" containerName="test-curl-7480-140695933441328"
Jul 31 15:13:44.844925 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.844915    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="808cab79-ef01-46ad-a3a7-1bb36567c4c2" containerName="test-curl-7480-140695933441328"
Jul 31 15:13:44.848124 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.848109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:44.851992 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.851968    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:44.897486 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.897460    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdnjk\" (UniqueName: \"kubernetes.io/projected/372cf03b-2fc0-4470-afbe-c1223629ed40-kube-api-access-mdnjk\") pod \"test-curl-7480-140695933450112\" (UID: \"372cf03b-2fc0-4470-afbe-c1223629ed40\") " pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:44.998246 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:44.998214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdnjk\" (UniqueName: \"kubernetes.io/projected/372cf03b-2fc0-4470-afbe-c1223629ed40-kube-api-access-mdnjk\") pod \"test-curl-7480-140695933450112\" (UID: \"372cf03b-2fc0-4470-afbe-c1223629ed40\") " pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:45.007044 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:45.006999    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdnjk\" (UniqueName: \"kubernetes.io/projected/372cf03b-2fc0-4470-afbe-c1223629ed40-kube-api-access-mdnjk\") pod \"test-curl-7480-140695933450112\" (UID: \"372cf03b-2fc0-4470-afbe-c1223629ed40\") " pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:45.159749 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:45.159662    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:45.301885 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:45.301856    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:45.303506 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:13:45.303478    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod372cf03b_2fc0_4470_afbe_c1223629ed40.slice/crio-d9c3358523d3284ab2960108b854a23d978732f00360117c52924f5fe3b772e9 WatchSource:0}: Error finding container d9c3358523d3284ab2960108b854a23d978732f00360117c52924f5fe3b772e9: Status 404 returned error can't find the container with id d9c3358523d3284ab2960108b854a23d978732f00360117c52924f5fe3b772e9
Jul 31 15:13:45.478006 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:45.477914    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933450112" event={"ID":"372cf03b-2fc0-4470-afbe-c1223629ed40","Type":"ContainerStarted","Data":"d9c3358523d3284ab2960108b854a23d978732f00360117c52924f5fe3b772e9"}
Jul 31 15:13:45.479077 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:45.479057    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="473b9b36d82e3efbefd19baf31d7180d2721a887ecb2a3b13e464f5241045779"
Jul 31 15:13:45.479198 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:45.479080    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933441328"
Jul 31 15:13:45.489929 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:13:45.489903    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7480-140695933441328\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="808cab79-ef01-46ad-a3a7-1bb36567c4c2" pod="opendatahub/test-curl-7480-140695933441328"
Jul 31 15:13:45.586167 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:45.586131    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="808cab79-ef01-46ad-a3a7-1bb36567c4c2" path="/var/lib/kubelet/pods/808cab79-ef01-46ad-a3a7-1bb36567c4c2/volumes"
Jul 31 15:13:46.486581 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:46.486543    2558 generic.go:356] "Generic (PLEG): container finished" podID="372cf03b-2fc0-4470-afbe-c1223629ed40" containerID="db016ae7c227f5411f9545781a5cbd1c78917917be5993381eb10e0cbb513b15" exitCode=0
Jul 31 15:13:46.486970 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:46.486625    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933450112" event={"ID":"372cf03b-2fc0-4470-afbe-c1223629ed40","Type":"ContainerDied","Data":"db016ae7c227f5411f9545781a5cbd1c78917917be5993381eb10e0cbb513b15"}
Jul 31 15:13:47.639140 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.639119    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:47.660243 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.660216    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7480-140695933450112_372cf03b-2fc0-4470-afbe-c1223629ed40/test-curl-7480-140695933450112/0.log"
Jul 31 15:13:47.684533 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.684498    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:47.686512 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.686490    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:47.722896 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.722869    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdnjk\" (UniqueName: \"kubernetes.io/projected/372cf03b-2fc0-4470-afbe-c1223629ed40-kube-api-access-mdnjk\") pod \"372cf03b-2fc0-4470-afbe-c1223629ed40\" (UID: \"372cf03b-2fc0-4470-afbe-c1223629ed40\") "
Jul 31 15:13:47.724856 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.724831    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372cf03b-2fc0-4470-afbe-c1223629ed40-kube-api-access-mdnjk" (OuterVolumeSpecName: "kube-api-access-mdnjk") pod "372cf03b-2fc0-4470-afbe-c1223629ed40" (UID: "372cf03b-2fc0-4470-afbe-c1223629ed40"). InnerVolumeSpecName "kube-api-access-mdnjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:47.823800 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.823774    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdnjk\" (UniqueName: \"kubernetes.io/projected/372cf03b-2fc0-4470-afbe-c1223629ed40-kube-api-access-mdnjk\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:47.828626 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.828599    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:47.829059 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.829045    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372cf03b-2fc0-4470-afbe-c1223629ed40" containerName="test-curl-7480-140695933450112"
Jul 31 15:13:47.829107 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.829061    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="372cf03b-2fc0-4470-afbe-c1223629ed40" containerName="test-curl-7480-140695933450112"
Jul 31 15:13:47.829154 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.829145    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="372cf03b-2fc0-4470-afbe-c1223629ed40" containerName="test-curl-7480-140695933450112"
Jul 31 15:13:47.832203 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.832183    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:47.840501 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.840477    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:47.925245 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:47.925212    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzzd\" (UniqueName: \"kubernetes.io/projected/e8634dd9-7aa2-4c90-8a5d-2b4a36615276-kube-api-access-7gzzd\") pod \"test-curl-7480-140695933450112\" (UID: \"e8634dd9-7aa2-4c90-8a5d-2b4a36615276\") " pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:48.026490 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:48.026453    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzzd\" (UniqueName: \"kubernetes.io/projected/e8634dd9-7aa2-4c90-8a5d-2b4a36615276-kube-api-access-7gzzd\") pod \"test-curl-7480-140695933450112\" (UID: \"e8634dd9-7aa2-4c90-8a5d-2b4a36615276\") " pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:48.034128 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:48.034107    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzzd\" (UniqueName: \"kubernetes.io/projected/e8634dd9-7aa2-4c90-8a5d-2b4a36615276-kube-api-access-7gzzd\") pod \"test-curl-7480-140695933450112\" (UID: \"e8634dd9-7aa2-4c90-8a5d-2b4a36615276\") " pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:48.142452 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:48.142367    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:48.278886 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:48.278861    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:48.280330 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:13:48.280297    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8634dd9_7aa2_4c90_8a5d_2b4a36615276.slice/crio-a8392c4913f2ee5717a3725f18b908f7df23c74717bf3454a8e7e2978532c7f5 WatchSource:0}: Error finding container a8392c4913f2ee5717a3725f18b908f7df23c74717bf3454a8e7e2978532c7f5: Status 404 returned error can't find the container with id a8392c4913f2ee5717a3725f18b908f7df23c74717bf3454a8e7e2978532c7f5
Jul 31 15:13:48.499418 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:48.499334    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:48.499418 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:48.499356    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c3358523d3284ab2960108b854a23d978732f00360117c52924f5fe3b772e9"
Jul 31 15:13:48.500602 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:48.500570    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933450112" event={"ID":"e8634dd9-7aa2-4c90-8a5d-2b4a36615276","Type":"ContainerStarted","Data":"a8392c4913f2ee5717a3725f18b908f7df23c74717bf3454a8e7e2978532c7f5"}
Jul 31 15:13:48.512222 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:48.512195    2558 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7480-140695933450112" oldPodUID="372cf03b-2fc0-4470-afbe-c1223629ed40" podUID="e8634dd9-7aa2-4c90-8a5d-2b4a36615276"
Jul 31 15:13:49.508628 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:49.508599    2558 generic.go:356] "Generic (PLEG): container finished" podID="e8634dd9-7aa2-4c90-8a5d-2b4a36615276" containerID="f78fb8d24c42bdfd084166dfa1a26ae74128cda537c8b034ffd59b85b53809d6" exitCode=0
Jul 31 15:13:49.508994 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:49.508673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933450112" event={"ID":"e8634dd9-7aa2-4c90-8a5d-2b4a36615276","Type":"ContainerDied","Data":"f78fb8d24c42bdfd084166dfa1a26ae74128cda537c8b034ffd59b85b53809d6"}
Jul 31 15:13:49.580337 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:49.580303    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="372cf03b-2fc0-4470-afbe-c1223629ed40" path="/var/lib/kubelet/pods/372cf03b-2fc0-4470-afbe-c1223629ed40/volumes"
Jul 31 15:13:50.658182 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.658155    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:50.680037 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.679998    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7480-140695933450112_e8634dd9-7aa2-4c90-8a5d-2b4a36615276/test-curl-7480-140695933450112/0.log"
Jul 31 15:13:50.704609 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.704582    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:50.707834 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.707803    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7480-140695933450112"]
Jul 31 15:13:50.754152 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.754118    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gzzd\" (UniqueName: \"kubernetes.io/projected/e8634dd9-7aa2-4c90-8a5d-2b4a36615276-kube-api-access-7gzzd\") pod \"e8634dd9-7aa2-4c90-8a5d-2b4a36615276\" (UID: \"e8634dd9-7aa2-4c90-8a5d-2b4a36615276\") "
Jul 31 15:13:50.756342 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.756315    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8634dd9-7aa2-4c90-8a5d-2b4a36615276-kube-api-access-7gzzd" (OuterVolumeSpecName: "kube-api-access-7gzzd") pod "e8634dd9-7aa2-4c90-8a5d-2b4a36615276" (UID: "e8634dd9-7aa2-4c90-8a5d-2b4a36615276"). InnerVolumeSpecName "kube-api-access-7gzzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:50.842699 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.842668    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:50.843163 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.843147    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8634dd9-7aa2-4c90-8a5d-2b4a36615276" containerName="test-curl-7480-140695933450112"
Jul 31 15:13:50.843207 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.843165    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8634dd9-7aa2-4c90-8a5d-2b4a36615276" containerName="test-curl-7480-140695933450112"
Jul 31 15:13:50.843249 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.843239    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8634dd9-7aa2-4c90-8a5d-2b4a36615276" containerName="test-curl-7480-140695933450112"
Jul 31 15:13:50.846309 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.846292    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:50.852863 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.852837    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:50.855390 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.855351    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzzd\" (UniqueName: \"kubernetes.io/projected/e8634dd9-7aa2-4c90-8a5d-2b4a36615276-kube-api-access-7gzzd\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:50.956395 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:50.956339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktm6w\" (UniqueName: \"kubernetes.io/projected/da576119-4f2b-4cdc-8eec-afafe72f7973-kube-api-access-ktm6w\") pod \"test-curl-7480-140695933451120\" (UID: \"da576119-4f2b-4cdc-8eec-afafe72f7973\") " pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:51.057737 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:51.057681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktm6w\" (UniqueName: \"kubernetes.io/projected/da576119-4f2b-4cdc-8eec-afafe72f7973-kube-api-access-ktm6w\") pod \"test-curl-7480-140695933451120\" (UID: \"da576119-4f2b-4cdc-8eec-afafe72f7973\") " pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:51.065929 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:51.065898    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktm6w\" (UniqueName: \"kubernetes.io/projected/da576119-4f2b-4cdc-8eec-afafe72f7973-kube-api-access-ktm6w\") pod \"test-curl-7480-140695933451120\" (UID: \"da576119-4f2b-4cdc-8eec-afafe72f7973\") " pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:51.158772 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:51.158685    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:51.295347 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:51.295320    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:51.297171 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:13:51.297144    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda576119_4f2b_4cdc_8eec_afafe72f7973.slice/crio-58265612395ed4b5e765441ca0fb7001df63a3a73b784c90bd2fcef6c609026c WatchSource:0}: Error finding container 58265612395ed4b5e765441ca0fb7001df63a3a73b784c90bd2fcef6c609026c: Status 404 returned error can't find the container with id 58265612395ed4b5e765441ca0fb7001df63a3a73b784c90bd2fcef6c609026c
Jul 31 15:13:51.520761 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:51.520671    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:51.520761 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:51.520676    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8392c4913f2ee5717a3725f18b908f7df23c74717bf3454a8e7e2978532c7f5"
Jul 31 15:13:51.521988 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:51.521961    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933451120" event={"ID":"da576119-4f2b-4cdc-8eec-afafe72f7973","Type":"ContainerStarted","Data":"58265612395ed4b5e765441ca0fb7001df63a3a73b784c90bd2fcef6c609026c"}
Jul 31 15:13:51.532353 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:13:51.532325    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7480-140695933450112\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="e8634dd9-7aa2-4c90-8a5d-2b4a36615276" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:51.579894 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:13:51.579865    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7480-140695933450112\" is forbidden: User \"system:node:ip-10-0-139-143.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-143.ec2.internal' and this object" podUID="e8634dd9-7aa2-4c90-8a5d-2b4a36615276" pod="opendatahub/test-curl-7480-140695933450112"
Jul 31 15:13:51.582664 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:51.582641    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8634dd9-7aa2-4c90-8a5d-2b4a36615276" path="/var/lib/kubelet/pods/e8634dd9-7aa2-4c90-8a5d-2b4a36615276/volumes"
Jul 31 15:13:52.212531 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.212483    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r" podUID="ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" containerName="main" containerID="cri-o://00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571" gracePeriod=45
Jul 31 15:13:52.486189 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.486165    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:13:52.530777 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.530743    2558 generic.go:356] "Generic (PLEG): container finished" podID="ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" containerID="00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571" exitCode=0
Jul 31 15:13:52.530952 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.530792    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"
Jul 31 15:13:52.530952 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.530831    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r" event={"ID":"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b","Type":"ContainerDied","Data":"00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571"}
Jul 31 15:13:52.530952 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.530881    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r" event={"ID":"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b","Type":"ContainerDied","Data":"c5a2f3fa5b42e37c1aae3ef52141604567d0f9369edf8fbf012964ddda104650"}
Jul 31 15:13:52.530952 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.530904    2558 scope.go:117] "RemoveContainer" containerID="00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571"
Jul 31 15:13:52.532690 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.532660    2558 generic.go:356] "Generic (PLEG): container finished" podID="da576119-4f2b-4cdc-8eec-afafe72f7973" containerID="ca891e3d8b6a26b81abab672e724dd011e37102330e11ce2f0a4ee72888d0c81" exitCode=0
Jul 31 15:13:52.532795 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.532747    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933451120" event={"ID":"da576119-4f2b-4cdc-8eec-afafe72f7973","Type":"ContainerDied","Data":"ca891e3d8b6a26b81abab672e724dd011e37102330e11ce2f0a4ee72888d0c81"}
Jul 31 15:13:52.540262 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.540243    2558 scope.go:117] "RemoveContainer" containerID="00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571"
Jul 31 15:13:52.540459 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:13:52.540442    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571\": container with ID starting with 00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571 not found: ID does not exist" containerID="00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571"
Jul 31 15:13:52.540523 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.540471    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571"} err="failed to get container status \"00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571\": rpc error: code = NotFound desc = could not find container \"00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571\": container with ID starting with 00e1683dc48c47503e9e6b208138176ed6179f88716dcf843117a888250ed571 not found: ID does not exist"
Jul 31 15:13:52.571676 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.571649    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-dshm\") pod \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") "
Jul 31 15:13:52.571845 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.571716    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-home\") pod \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") "
Jul 31 15:13:52.571845 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.571744    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tls-certs\") pod \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") "
Jul 31 15:13:52.571845 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.571799    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9h4x\" (UniqueName: \"kubernetes.io/projected/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-kube-api-access-h9h4x\") pod \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") "
Jul 31 15:13:52.571845 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.571831    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tmp-dir\") pod \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") "
Jul 31 15:13:52.572069 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.571890    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-model-cache\") pod \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\" (UID: \"ee39e5a3-11be-4dde-ac8a-30d3cf6e070b\") "
Jul 31 15:13:52.572069 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.571997    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-home" (OuterVolumeSpecName: "home") pod "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" (UID: "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:13:52.572198 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.572134    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" (UID: "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:13:52.572252 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.572186    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" (UID: "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:13:52.572310 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.572294    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:52.572343 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.572317    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:52.572343 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.572331    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:52.573896 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.573875    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-dshm" (OuterVolumeSpecName: "dshm") pod "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" (UID: "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:13:52.573998 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.573923    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" (UID: "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:13:52.573998 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.573956    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-kube-api-access-h9h4x" (OuterVolumeSpecName: "kube-api-access-h9h4x") pod "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" (UID: "ee39e5a3-11be-4dde-ac8a-30d3cf6e070b"). InnerVolumeSpecName "kube-api-access-h9h4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:52.673125 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.673093    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:52.673125 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.673120    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9h4x\" (UniqueName: \"kubernetes.io/projected/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-kube-api-access-h9h4x\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:52.673125 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.673129    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:52.852945 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.852912    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"]
Jul 31 15:13:52.854857 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:52.854839    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-79a3ee/rate-test-model-79a3ee-kserve-bfbcf7547-vnk2r"]
Jul 31 15:13:53.581553 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.581515    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" path="/var/lib/kubelet/pods/ee39e5a3-11be-4dde-ac8a-30d3cf6e070b/volumes"
Jul 31 15:13:53.703199 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.703170    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:53.724690 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.724654    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7480-140695933451120_da576119-4f2b-4cdc-8eec-afafe72f7973/test-curl-7480-140695933451120/0.log"
Jul 31 15:13:53.750282 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.750255    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:53.754700 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.754678    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:53.783570 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.783544    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktm6w\" (UniqueName: \"kubernetes.io/projected/da576119-4f2b-4cdc-8eec-afafe72f7973-kube-api-access-ktm6w\") pod \"da576119-4f2b-4cdc-8eec-afafe72f7973\" (UID: \"da576119-4f2b-4cdc-8eec-afafe72f7973\") "
Jul 31 15:13:53.785571 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.785539    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da576119-4f2b-4cdc-8eec-afafe72f7973-kube-api-access-ktm6w" (OuterVolumeSpecName: "kube-api-access-ktm6w") pod "da576119-4f2b-4cdc-8eec-afafe72f7973" (UID: "da576119-4f2b-4cdc-8eec-afafe72f7973"). InnerVolumeSpecName "kube-api-access-ktm6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:53.882181 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.882104    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:53.882569 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.882557    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" containerName="main"
Jul 31 15:13:53.882619 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.882572    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" containerName="main"
Jul 31 15:13:53.882619 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.882594    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da576119-4f2b-4cdc-8eec-afafe72f7973" containerName="test-curl-7480-140695933451120"
Jul 31 15:13:53.882619 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.882603    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="da576119-4f2b-4cdc-8eec-afafe72f7973" containerName="test-curl-7480-140695933451120"
Jul 31 15:13:53.882714 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.882663    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="da576119-4f2b-4cdc-8eec-afafe72f7973" containerName="test-curl-7480-140695933451120"
Jul 31 15:13:53.882714 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.882673    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee39e5a3-11be-4dde-ac8a-30d3cf6e070b" containerName="main"
Jul 31 15:13:53.885187 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.885166    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktm6w\" (UniqueName: \"kubernetes.io/projected/da576119-4f2b-4cdc-8eec-afafe72f7973-kube-api-access-ktm6w\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:53.885745 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.885731    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:53.892098 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.892072    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:53.986215 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:53.986165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lx8\" (UniqueName: \"kubernetes.io/projected/db752d18-e387-4c32-b710-a1d810adacce-kube-api-access-98lx8\") pod \"test-curl-7480-140695933451120\" (UID: \"db752d18-e387-4c32-b710-a1d810adacce\") " pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:54.087116 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:54.087080    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lx8\" (UniqueName: \"kubernetes.io/projected/db752d18-e387-4c32-b710-a1d810adacce-kube-api-access-98lx8\") pod \"test-curl-7480-140695933451120\" (UID: \"db752d18-e387-4c32-b710-a1d810adacce\") " pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:54.095568 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:54.095539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lx8\" (UniqueName: \"kubernetes.io/projected/db752d18-e387-4c32-b710-a1d810adacce-kube-api-access-98lx8\") pod \"test-curl-7480-140695933451120\" (UID: \"db752d18-e387-4c32-b710-a1d810adacce\") " pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:54.197306 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:54.197226    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:54.335947 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:54.335914    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:54.337750 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:13:54.337723    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb752d18_e387_4c32_b710_a1d810adacce.slice/crio-3cf3f9766e2480777caae61ded5cb67bf8a10bd6dcd96e39bc86f48a70fe2721 WatchSource:0}: Error finding container 3cf3f9766e2480777caae61ded5cb67bf8a10bd6dcd96e39bc86f48a70fe2721: Status 404 returned error can't find the container with id 3cf3f9766e2480777caae61ded5cb67bf8a10bd6dcd96e39bc86f48a70fe2721
Jul 31 15:13:54.554783 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:54.554750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933451120" event={"ID":"db752d18-e387-4c32-b710-a1d810adacce","Type":"ContainerStarted","Data":"3cf3f9766e2480777caae61ded5cb67bf8a10bd6dcd96e39bc86f48a70fe2721"}
Jul 31 15:13:54.556034 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:54.556001    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:54.556162 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:54.556044    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58265612395ed4b5e765441ca0fb7001df63a3a73b784c90bd2fcef6c609026c"
Jul 31 15:13:54.568298 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:54.568269    2558 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7480-140695933451120" oldPodUID="da576119-4f2b-4cdc-8eec-afafe72f7973" podUID="db752d18-e387-4c32-b710-a1d810adacce"
Jul 31 15:13:55.563965 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:55.563930    2558 generic.go:356] "Generic (PLEG): container finished" podID="db752d18-e387-4c32-b710-a1d810adacce" containerID="9aec9942413e526244c33b5b552e1357aa002f73f3386a17a2bbdc8f49e03b73" exitCode=0
Jul 31 15:13:55.564426 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:55.564018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7480-140695933451120" event={"ID":"db752d18-e387-4c32-b710-a1d810adacce","Type":"ContainerDied","Data":"9aec9942413e526244c33b5b552e1357aa002f73f3386a17a2bbdc8f49e03b73"}
Jul 31 15:13:55.579840 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:55.579808    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da576119-4f2b-4cdc-8eec-afafe72f7973" path="/var/lib/kubelet/pods/da576119-4f2b-4cdc-8eec-afafe72f7973/volumes"
Jul 31 15:13:56.714817 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:56.714795    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:56.734833 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:56.734807    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7480-140695933451120_db752d18-e387-4c32-b710-a1d810adacce/test-curl-7480-140695933451120/0.log"
Jul 31 15:13:56.758773 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:56.758742    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:56.760829 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:56.760807    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7480-140695933451120"]
Jul 31 15:13:56.811932 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:56.811900    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98lx8\" (UniqueName: \"kubernetes.io/projected/db752d18-e387-4c32-b710-a1d810adacce-kube-api-access-98lx8\") pod \"db752d18-e387-4c32-b710-a1d810adacce\" (UID: \"db752d18-e387-4c32-b710-a1d810adacce\") "
Jul 31 15:13:56.814010 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:56.813987    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db752d18-e387-4c32-b710-a1d810adacce-kube-api-access-98lx8" (OuterVolumeSpecName: "kube-api-access-98lx8") pod "db752d18-e387-4c32-b710-a1d810adacce" (UID: "db752d18-e387-4c32-b710-a1d810adacce"). InnerVolumeSpecName "kube-api-access-98lx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:56.913203 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:56.913117    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lx8\" (UniqueName: \"kubernetes.io/projected/db752d18-e387-4c32-b710-a1d810adacce-kube-api-access-98lx8\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:13:57.577472 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:57.577441    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7480-140695933451120"
Jul 31 15:13:57.580536 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:57.580502    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db752d18-e387-4c32-b710-a1d810adacce" path="/var/lib/kubelet/pods/db752d18-e387-4c32-b710-a1d810adacce/volumes"
Jul 31 15:13:57.581077 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:13:57.581058    2558 scope.go:117] "RemoveContainer" containerID="9aec9942413e526244c33b5b552e1357aa002f73f3386a17a2bbdc8f49e03b73"
Jul 31 15:14:26.751048 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.750991    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"]
Jul 31 15:14:26.752129 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.751697    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db752d18-e387-4c32-b710-a1d810adacce" containerName="test-curl-7480-140695933451120"
Jul 31 15:14:26.752129 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.751719    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="db752d18-e387-4c32-b710-a1d810adacce" containerName="test-curl-7480-140695933451120"
Jul 31 15:14:26.752129 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.751844    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="db752d18-e387-4c32-b710-a1d810adacce" containerName="test-curl-7480-140695933451120"
Jul 31 15:14:26.755112 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.755088    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.757737 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.757717    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1c9d21-openshift-default-dockercfg-94zbp\""
Jul 31 15:14:26.765214 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.765192    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"]
Jul 31 15:14:26.797529 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797498    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-credential-socket\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.797690 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797550    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-data\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.797690 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-socket\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.797690 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797679    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-certs\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.797826 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797746    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-envoy\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.797826 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797771    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-token\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.797826 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9fl\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-kube-api-access-pm9fl\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.797826 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797823    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-podinfo\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.798010 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.797934    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istiod-ca-cert\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899197 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899156    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-envoy\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899197 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899197    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-token\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899467 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899224    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9fl\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-kube-api-access-pm9fl\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899467 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899259    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-podinfo\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899467 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istiod-ca-cert\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899467 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-credential-socket\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899467 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899439    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-data\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899742 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-socket\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899742 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-certs\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899838 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899755    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-credential-socket\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.899884 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.899872    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-data\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.900045 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.900007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-certs\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.900214 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.900189    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istiod-ca-cert\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.900495 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.900470    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-socket\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.901699 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.901674    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-envoy\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.901929 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.901909    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-podinfo\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.907586 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.907562    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-token\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:26.907874 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:26.907855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9fl\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-kube-api-access-pm9fl\") pod \"e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") " pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:27.067964 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:27.067926    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:27.213365 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:27.213335    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"]
Jul 31 15:14:27.214904 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:14:27.214869    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf695c4_4163_4e4d_9bd9_7180543b9e1a.slice/crio-6f63472579ee6d82f8755ab41a510cd788082e746ac742b9380b17dd5e3372b9 WatchSource:0}: Error finding container 6f63472579ee6d82f8755ab41a510cd788082e746ac742b9380b17dd5e3372b9: Status 404 returned error can't find the container with id 6f63472579ee6d82f8755ab41a510cd788082e746ac742b9380b17dd5e3372b9
Jul 31 15:14:27.217306 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:27.217265    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:14:27.217396 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:27.217358    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:14:27.217436 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:27.217401    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:14:27.752531 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:27.752497    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" event={"ID":"bcf695c4-4163-4e4d-9bd9-7180543b9e1a","Type":"ContainerStarted","Data":"34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4"}
Jul 31 15:14:27.752924 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:27.752553    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" event={"ID":"bcf695c4-4163-4e4d-9bd9-7180543b9e1a","Type":"ContainerStarted","Data":"6f63472579ee6d82f8755ab41a510cd788082e746ac742b9380b17dd5e3372b9"}
Jul 31 15:14:27.774299 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:27.774256    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podStartSLOduration=1.774241234 podStartE2EDuration="1.774241234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:14:27.774194976 +0000 UTC m=+3706.646758231" watchObservedRunningTime="2026-07-31 15:14:27.774241234 +0000 UTC m=+3706.646804485"
Jul 31 15:14:28.068607 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:28.068573    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:28.073808 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:28.073785    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:28.758185 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:28.758155    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:28.759270 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:28.759248    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:43.927660 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:43.927618    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": read tcp 10.134.0.2:47370->10.134.0.82:15021: read: connection reset by peer" start-of-body=
Jul 31 15:14:43.928078 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:43.927690    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": read tcp 10.134.0.2:47370->10.134.0.82:15021: read: connection reset by peer"
Jul 31 15:14:44.853474 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:44.853442    2558 generic.go:356] "Generic (PLEG): container finished" podID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerID="34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4" exitCode=0
Jul 31 15:14:44.853697 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:44.853527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" event={"ID":"bcf695c4-4163-4e4d-9bd9-7180543b9e1a","Type":"ContainerDied","Data":"34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4"}
Jul 31 15:14:44.854004 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:44.853985    2558 scope.go:117] "RemoveContainer" containerID="34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4"
Jul 31 15:14:44.855130 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:44.855101    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:14:44.855196 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:44.855174    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:14:44.855239 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:44.855213    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:14:45.860821 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:45.860777    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" event={"ID":"bcf695c4-4163-4e4d-9bd9-7180543b9e1a","Type":"ContainerStarted","Data":"c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d"}
Jul 31 15:14:46.068785 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:46.068748    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:46.068989 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:46.068965    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Jul 31 15:14:46.069082 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:46.069012    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Jul 31 15:14:47.068765 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:47.068725    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Jul 31 15:14:47.069269 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:47.068788    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Jul 31 15:14:48.068451 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:48.068412    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Jul 31 15:14:48.068674 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:48.068478    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Jul 31 15:14:49.069331 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:49.069290    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Jul 31 15:14:49.069823 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:49.069357    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Jul 31 15:14:50.068338 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:50.068305    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Jul 31 15:14:50.068516 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:50.068366    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Jul 31 15:14:51.069239 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:51.069205    2558 patch_prober.go:28] interesting pod/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Jul 31 15:14:51.069658 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:51.069261    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Jul 31 15:14:52.071936 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:52.071908    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:52.072437 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:52.072318    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:14:52.073034 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:14:52.073002    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:15:00.141439 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.141404    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"]
Jul 31 15:15:00.153867 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.153800    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"]
Jul 31 15:15:00.154409 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.153937    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"
Jul 31 15:15:00.157971 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.157859    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:15:00.159900 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.159879    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd"]
Jul 31 15:15:00.161476 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.160001    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"
Jul 31 15:15:00.166563 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.164098    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"]
Jul 31 15:15:00.166563 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.165717    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd"
Jul 31 15:15:00.168203 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.168165    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"]
Jul 31 15:15:00.168625 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.168440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"
Jul 31 15:15:00.172359 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.172340    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"
Jul 31 15:15:00.172601 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.172585    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"]
Jul 31 15:15:00.172673 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.172622    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"]
Jul 31 15:15:00.172734 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.172691    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"
Jul 31 15:15:00.172791 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.172739    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"]
Jul 31 15:15:00.174823 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.174797    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd"]
Jul 31 15:15:00.177919 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.177902    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"]
Jul 31 15:15:00.180842 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.180825    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"
Jul 31 15:15:00.220276 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.220249    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd"
Jul 31 15:15:00.230567 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.230541    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"
Jul 31 15:15:00.280038 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.279830    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"
Jul 31 15:15:00.405603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.405473    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"]
Jul 31 15:15:00.410324 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:15:00.410278    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21f4200_a559_4dd2_9711_90bbd84bfdb3.slice/crio-55e09987a34e1d009acde489c68c7e6bf61d1ad3591d240cc22768f7177861fb WatchSource:0}: Error finding container 55e09987a34e1d009acde489c68c7e6bf61d1ad3591d240cc22768f7177861fb: Status 404 returned error can't find the container with id 55e09987a34e1d009acde489c68c7e6bf61d1ad3591d240cc22768f7177861fb
Jul 31 15:15:00.435775 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.435735    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd"]
Jul 31 15:15:00.464683 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.464658    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"]
Jul 31 15:15:00.474512 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:15:00.474483    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2c8b54_846a_4b59_b5dd_0f806ac5a810.slice/crio-df69093978b15397b57fdbe7f6ddedcfefb52db144aad6ecce7efe7a4ec30245 WatchSource:0}: Error finding container df69093978b15397b57fdbe7f6ddedcfefb52db144aad6ecce7efe7a4ec30245: Status 404 returned error can't find the container with id df69093978b15397b57fdbe7f6ddedcfefb52db144aad6ecce7efe7a4ec30245
Jul 31 15:15:00.493766 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.493742    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"]
Jul 31 15:15:00.496139 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:15:00.496114    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845c58ca_85f3_4313_88f7_8ac226431e4a.slice/crio-92c7892a28d5eba9de16aab0528545e9fd81454e174606571976c8cb7f860366 WatchSource:0}: Error finding container 92c7892a28d5eba9de16aab0528545e9fd81454e174606571976c8cb7f860366: Status 404 returned error can't find the container with id 92c7892a28d5eba9de16aab0528545e9fd81454e174606571976c8cb7f860366
Jul 31 15:15:00.559387 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.559364    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"]
Jul 31 15:15:00.560826 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:15:00.560799    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594bdf05_b917_42c1_87bb_6e8995c0596c.slice/crio-161dd1fa1fa99190c3b36e6a04f2e100688cc92a721a9b0b62833938d09bc7a0 WatchSource:0}: Error finding container 161dd1fa1fa99190c3b36e6a04f2e100688cc92a721a9b0b62833938d09bc7a0: Status 404 returned error can't find the container with id 161dd1fa1fa99190c3b36e6a04f2e100688cc92a721a9b0b62833938d09bc7a0
Jul 31 15:15:00.946802 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.946772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz" event={"ID":"ef2c8b54-846a-4b59-b5dd-0f806ac5a810","Type":"ContainerStarted","Data":"39d8bdff3303cb213f1f6aea455f442d28dcab49eccb5e34b2656f0da2404145"}
Jul 31 15:15:00.946927 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.946828    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz" event={"ID":"ef2c8b54-846a-4b59-b5dd-0f806ac5a810","Type":"ContainerStarted","Data":"df69093978b15397b57fdbe7f6ddedcfefb52db144aad6ecce7efe7a4ec30245"}
Jul 31 15:15:00.948309 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.948283    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt" event={"ID":"594bdf05-b917-42c1-87bb-6e8995c0596c","Type":"ContainerStarted","Data":"f5ab18dcc2e912f04257a4e09b8f1b87eebb244dee1ad470c2e61542c6f4129b"}
Jul 31 15:15:00.948430 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.948329    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt" event={"ID":"594bdf05-b917-42c1-87bb-6e8995c0596c","Type":"ContainerStarted","Data":"161dd1fa1fa99190c3b36e6a04f2e100688cc92a721a9b0b62833938d09bc7a0"}
Jul 31 15:15:00.949746 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.949719    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s" event={"ID":"e21f4200-a559-4dd2-9711-90bbd84bfdb3","Type":"ContainerStarted","Data":"a6a5d3f25b6cf2c1e1907f369ecfba4bc3cb09692e8b41f8008dbc50349a5cbb"}
Jul 31 15:15:00.949861 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.949763    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s" event={"ID":"e21f4200-a559-4dd2-9711-90bbd84bfdb3","Type":"ContainerStarted","Data":"55e09987a34e1d009acde489c68c7e6bf61d1ad3591d240cc22768f7177861fb"}
Jul 31 15:15:00.951189 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.951165    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd" event={"ID":"6e4680bf-d449-4c90-aabc-0d2e8d07e4e4","Type":"ContainerStarted","Data":"59d3b0b4afe40bf67136d5767511e27ad5c5322cf8282046762704ae0b68d473"}
Jul 31 15:15:00.951304 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.951212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd" event={"ID":"6e4680bf-d449-4c90-aabc-0d2e8d07e4e4","Type":"ContainerStarted","Data":"9b4c8bab4c39a1a5029aef641d3000f0153e78c90af1b6a20c2add4b35493175"}
Jul 31 15:15:00.952552 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.952531    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm" event={"ID":"845c58ca-85f3-4313-88f7-8ac226431e4a","Type":"ContainerStarted","Data":"0c403567d6f393ae287d3d344ab9cc4d31870eb8223f71dac206fa8af14b0bee"}
Jul 31 15:15:00.952634 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.952566    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm" event={"ID":"845c58ca-85f3-4313-88f7-8ac226431e4a","Type":"ContainerStarted","Data":"92c7892a28d5eba9de16aab0528545e9fd81454e174606571976c8cb7f860366"}
Jul 31 15:15:00.962227 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.962190    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz" podStartSLOduration=0.962177289 podStartE2EDuration="962.177289ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:00.959927628 +0000 UTC m=+3739.832490882" watchObservedRunningTime="2026-07-31 15:15:00.962177289 +0000 UTC m=+3739.834740541"
Jul 31 15:15:00.973882 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.973838    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s" podStartSLOduration=0.973826103 podStartE2EDuration="973.826103ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:00.97292226 +0000 UTC m=+3739.845485510" watchObservedRunningTime="2026-07-31 15:15:00.973826103 +0000 UTC m=+3739.846389356"
Jul 31 15:15:00.986456 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.986383    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd" podStartSLOduration=0.986371732 podStartE2EDuration="986.371732ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:00.985354705 +0000 UTC m=+3739.857917958" watchObservedRunningTime="2026-07-31 15:15:00.986371732 +0000 UTC m=+3739.858934997"
Jul 31 15:15:00.999180 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:00.999147    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm" podStartSLOduration=0.999138035 podStartE2EDuration="999.138035ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:00.99858759 +0000 UTC m=+3739.871150844" watchObservedRunningTime="2026-07-31 15:15:00.999138035 +0000 UTC m=+3739.871701287"
Jul 31 15:15:01.012560 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.012514    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt" podStartSLOduration=1.012497556 podStartE2EDuration="1.012497556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:01.011093967 +0000 UTC m=+3739.883657219" watchObservedRunningTime="2026-07-31 15:15:01.012497556 +0000 UTC m=+3739.885060809"
Jul 31 15:15:01.960317 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.960277    2558 generic.go:356] "Generic (PLEG): container finished" podID="e21f4200-a559-4dd2-9711-90bbd84bfdb3" containerID="a6a5d3f25b6cf2c1e1907f369ecfba4bc3cb09692e8b41f8008dbc50349a5cbb" exitCode=0
Jul 31 15:15:01.960752 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.960360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s" event={"ID":"e21f4200-a559-4dd2-9711-90bbd84bfdb3","Type":"ContainerDied","Data":"a6a5d3f25b6cf2c1e1907f369ecfba4bc3cb09692e8b41f8008dbc50349a5cbb"}
Jul 31 15:15:01.961838 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.961817    2558 generic.go:356] "Generic (PLEG): container finished" podID="6e4680bf-d449-4c90-aabc-0d2e8d07e4e4" containerID="59d3b0b4afe40bf67136d5767511e27ad5c5322cf8282046762704ae0b68d473" exitCode=0
Jul 31 15:15:01.961928 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.961913    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd" event={"ID":"6e4680bf-d449-4c90-aabc-0d2e8d07e4e4","Type":"ContainerDied","Data":"59d3b0b4afe40bf67136d5767511e27ad5c5322cf8282046762704ae0b68d473"}
Jul 31 15:15:01.963547 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.963521    2558 generic.go:356] "Generic (PLEG): container finished" podID="845c58ca-85f3-4313-88f7-8ac226431e4a" containerID="0c403567d6f393ae287d3d344ab9cc4d31870eb8223f71dac206fa8af14b0bee" exitCode=0
Jul 31 15:15:01.963659 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.963595    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm" event={"ID":"845c58ca-85f3-4313-88f7-8ac226431e4a","Type":"ContainerDied","Data":"0c403567d6f393ae287d3d344ab9cc4d31870eb8223f71dac206fa8af14b0bee"}
Jul 31 15:15:01.965263 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.965241    2558 generic.go:356] "Generic (PLEG): container finished" podID="ef2c8b54-846a-4b59-b5dd-0f806ac5a810" containerID="39d8bdff3303cb213f1f6aea455f442d28dcab49eccb5e34b2656f0da2404145" exitCode=0
Jul 31 15:15:01.965346 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.965319    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz" event={"ID":"ef2c8b54-846a-4b59-b5dd-0f806ac5a810","Type":"ContainerDied","Data":"39d8bdff3303cb213f1f6aea455f442d28dcab49eccb5e34b2656f0da2404145"}
Jul 31 15:15:01.966815 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.966794    2558 generic.go:356] "Generic (PLEG): container finished" podID="594bdf05-b917-42c1-87bb-6e8995c0596c" containerID="f5ab18dcc2e912f04257a4e09b8f1b87eebb244dee1ad470c2e61542c6f4129b" exitCode=0
Jul 31 15:15:01.966911 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:01.966838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt" event={"ID":"594bdf05-b917-42c1-87bb-6e8995c0596c","Type":"ContainerDied","Data":"f5ab18dcc2e912f04257a4e09b8f1b87eebb244dee1ad470c2e61542c6f4129b"}
Jul 31 15:15:02.983161 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:02.983127    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"]
Jul 31 15:15:02.987842 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:02.987820    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:02.996471 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:02.995683    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"]
Jul 31 15:15:02.996471 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:02.995916    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-58af04\"/\"ipp-route-58af04-kserve-self-signed-certs\""
Jul 31 15:15:02.996471 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:02.996005    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-58af04\"/\"kube-root-ca.crt\""
Jul 31 15:15:02.996471 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:02.996168    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-58af04\"/\"openshift-service-ca.crt\""
Jul 31 15:15:02.996471 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:02.995918    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-58af04\"/\"default-dockercfg-qjbt6\""
Jul 31 15:15:03.062760 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.062381    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-dshm\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.062760 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.062452    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hvb\" (UniqueName: \"kubernetes.io/projected/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-kube-api-access-45hvb\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.062760 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.062505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tls-certs\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.062760 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.062553    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tmp-dir\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.062760 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.062647    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-model-cache\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.062760 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.062674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-home\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.148720 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.148175    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"
Jul 31 15:15:03.164123 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.163613    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tls-certs\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.164123 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.163690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tmp-dir\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.164123 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.163786    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-model-cache\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.164123 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.163814    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-home\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.164123 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.163868    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-dshm\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.164123 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.163916    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hvb\" (UniqueName: \"kubernetes.io/projected/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-kube-api-access-45hvb\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.168207 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.165039    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-model-cache\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.168207 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.165572    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tmp-dir\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.168207 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.165858    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-home\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.174628 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.171479    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-dshm\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.174628 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.173305    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tls-certs\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.181954 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.181696    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hvb\" (UniqueName: \"kubernetes.io/projected/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-kube-api-access-45hvb\") pod \"ipp-route-58af04-kserve-68ff6dfd79-652h9\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") " pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.250528 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.250500    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"
Jul 31 15:15:03.253094 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.253076    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"
Jul 31 15:15:03.256793 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.256774    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"
Jul 31 15:15:03.260310 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.260292    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd"
Jul 31 15:15:03.309207 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.309177    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:03.472192 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.472168    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"]
Jul 31 15:15:03.473308 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:15:03.473282    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4abbd14_9fa4_44c3_9383_3eda3cfd6718.slice/crio-45fe2ca612639252f8061f604b1952e5fdde2d75b8c65ec7eff861274537bc8a WatchSource:0}: Error finding container 45fe2ca612639252f8061f604b1952e5fdde2d75b8c65ec7eff861274537bc8a: Status 404 returned error can't find the container with id 45fe2ca612639252f8061f604b1952e5fdde2d75b8c65ec7eff861274537bc8a
Jul 31 15:15:03.991250 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.991216    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz" event={"ID":"ef2c8b54-846a-4b59-b5dd-0f806ac5a810","Type":"ContainerDied","Data":"df69093978b15397b57fdbe7f6ddedcfefb52db144aad6ecce7efe7a4ec30245"}
Jul 31 15:15:03.991683 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.991301    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df69093978b15397b57fdbe7f6ddedcfefb52db144aad6ecce7efe7a4ec30245"
Jul 31 15:15:03.991683 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.991227    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"
Jul 31 15:15:03.992557 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.992534    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt" event={"ID":"594bdf05-b917-42c1-87bb-6e8995c0596c","Type":"ContainerDied","Data":"161dd1fa1fa99190c3b36e6a04f2e100688cc92a721a9b0b62833938d09bc7a0"}
Jul 31 15:15:03.992557 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.992551    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"
Jul 31 15:15:03.992736 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.992568    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="161dd1fa1fa99190c3b36e6a04f2e100688cc92a721a9b0b62833938d09bc7a0"
Jul 31 15:15:03.994210 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.994179    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9" event={"ID":"f4abbd14-9fa4-44c3-9383-3eda3cfd6718","Type":"ContainerStarted","Data":"d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97"}
Jul 31 15:15:03.994315 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.994217    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9" event={"ID":"f4abbd14-9fa4-44c3-9383-3eda3cfd6718","Type":"ContainerStarted","Data":"45fe2ca612639252f8061f604b1952e5fdde2d75b8c65ec7eff861274537bc8a"}
Jul 31 15:15:03.995674 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.995657    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"
Jul 31 15:15:03.995674 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.995663    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s" event={"ID":"e21f4200-a559-4dd2-9711-90bbd84bfdb3","Type":"ContainerDied","Data":"55e09987a34e1d009acde489c68c7e6bf61d1ad3591d240cc22768f7177861fb"}
Jul 31 15:15:03.995802 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.995693    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55e09987a34e1d009acde489c68c7e6bf61d1ad3591d240cc22768f7177861fb"
Jul 31 15:15:03.997053 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.997016    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd"
Jul 31 15:15:03.997185 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.997049    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29758515-vb5qd" event={"ID":"6e4680bf-d449-4c90-aabc-0d2e8d07e4e4","Type":"ContainerDied","Data":"9b4c8bab4c39a1a5029aef641d3000f0153e78c90af1b6a20c2add4b35493175"}
Jul 31 15:15:03.997259 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.997215    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b4c8bab4c39a1a5029aef641d3000f0153e78c90af1b6a20c2add4b35493175"
Jul 31 15:15:03.998509 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.998494    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"
Jul 31 15:15:03.998616 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.998503    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm" event={"ID":"845c58ca-85f3-4313-88f7-8ac226431e4a","Type":"ContainerDied","Data":"92c7892a28d5eba9de16aab0528545e9fd81454e174606571976c8cb7f860366"}
Jul 31 15:15:03.998616 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:03.998538    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92c7892a28d5eba9de16aab0528545e9fd81454e174606571976c8cb7f860366"
Jul 31 15:15:04.013840 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:04.013788    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9" podStartSLOduration=2.013773117 podStartE2EDuration="2.013773117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:04.011004728 +0000 UTC m=+3742.883567984" watchObservedRunningTime="2026-07-31 15:15:04.013773117 +0000 UTC m=+3742.886336419"
Jul 31 15:15:04.346198 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:04.346163    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"]
Jul 31 15:15:04.350820 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:04.350792    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29758470-r95nr"]
Jul 31 15:15:05.580961 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:05.580930    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b30599-5e24-40a8-8f6f-548023e803be" path="/var/lib/kubelet/pods/67b30599-5e24-40a8-8f6f-548023e803be/volumes"
Jul 31 15:15:13.310000 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:13.309960    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:13.310596 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:13.310018    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:13.322814 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:13.322788    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:14.069427 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:14.069395    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:15:42.307937 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.307822    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"]
Jul 31 15:15:42.308476 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308460    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef2c8b54-846a-4b59-b5dd-0f806ac5a810" containerName="cleanup"
Jul 31 15:15:42.308528 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308480    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef2c8b54-846a-4b59-b5dd-0f806ac5a810" containerName="cleanup"
Jul 31 15:15:42.308528 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308510    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e21f4200-a559-4dd2-9711-90bbd84bfdb3" containerName="cleanup"
Jul 31 15:15:42.308528 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308519    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e21f4200-a559-4dd2-9711-90bbd84bfdb3" containerName="cleanup"
Jul 31 15:15:42.308631 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308529    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845c58ca-85f3-4313-88f7-8ac226431e4a" containerName="cleanup"
Jul 31 15:15:42.308631 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308537    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="845c58ca-85f3-4313-88f7-8ac226431e4a" containerName="cleanup"
Jul 31 15:15:42.308631 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308548    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="594bdf05-b917-42c1-87bb-6e8995c0596c" containerName="cleanup"
Jul 31 15:15:42.308631 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308557    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="594bdf05-b917-42c1-87bb-6e8995c0596c" containerName="cleanup"
Jul 31 15:15:42.308631 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308577    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e4680bf-d449-4c90-aabc-0d2e8d07e4e4" containerName="cleanup"
Jul 31 15:15:42.308631 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308587    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e4680bf-d449-4c90-aabc-0d2e8d07e4e4" containerName="cleanup"
Jul 31 15:15:42.308822 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308675    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="845c58ca-85f3-4313-88f7-8ac226431e4a" containerName="cleanup"
Jul 31 15:15:42.308822 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308688    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e21f4200-a559-4dd2-9711-90bbd84bfdb3" containerName="cleanup"
Jul 31 15:15:42.308822 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308702    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e4680bf-d449-4c90-aabc-0d2e8d07e4e4" containerName="cleanup"
Jul 31 15:15:42.308822 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308710    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="594bdf05-b917-42c1-87bb-6e8995c0596c" containerName="cleanup"
Jul 31 15:15:42.308822 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.308717    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef2c8b54-846a-4b59-b5dd-0f806ac5a810" containerName="cleanup"
Jul 31 15:15:42.313427 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.313403    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.316214 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.316192    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-39573e-openshift-default-dockercfg-lr95r\""
Jul 31 15:15:42.323975 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.323738    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"]
Jul 31 15:15:42.442464 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-envoy\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.442675 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-credential-socket\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.442675 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-data\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.442675 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442593    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-certs\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.442675 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-istio-token\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.442675 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19f1dea-795a-4047-a006-874adb74bef4-istio-podinfo\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.442954 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snt8x\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-kube-api-access-snt8x\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.442954 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442730    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19f1dea-795a-4047-a006-874adb74bef4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.442954 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.442769    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-socket\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.543621 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.543584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snt8x\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-kube-api-access-snt8x\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.543621 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.543622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19f1dea-795a-4047-a006-874adb74bef4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.543888 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.543753    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-socket\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.543888 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.543795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-envoy\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.543888 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.543880    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-credential-socket\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544081 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.543939    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-data\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544081 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.543974    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-certs\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544081 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.544008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-istio-token\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544260 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.544081    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19f1dea-795a-4047-a006-874adb74bef4-istio-podinfo\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544260 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.544181    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-socket\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544374 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.544258    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-credential-socket\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544374 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.544313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-certs\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544374 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.544364    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19f1dea-795a-4047-a006-874adb74bef4-istiod-ca-cert\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.544564 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.544545    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-data\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.546138 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.546105    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-envoy\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.546433 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.546414    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19f1dea-795a-4047-a006-874adb74bef4-istio-podinfo\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.551541 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.551521    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-istio-token\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.551760 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.551740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snt8x\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-kube-api-access-snt8x\") pod \"e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") " pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.627841 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.627764    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:42.775351 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.775323    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"]
Jul 31 15:15:42.776982 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:15:42.776954    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19f1dea_795a_4047_a006_874adb74bef4.slice/crio-9ef1019f52949ca0df1fb2b2d614fea4708ea8c06db81f9ee664449c8dafeb7e WatchSource:0}: Error finding container 9ef1019f52949ca0df1fb2b2d614fea4708ea8c06db81f9ee664449c8dafeb7e: Status 404 returned error can't find the container with id 9ef1019f52949ca0df1fb2b2d614fea4708ea8c06db81f9ee664449c8dafeb7e
Jul 31 15:15:42.779935 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.779905    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:15:42.780018 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.779972    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:15:42.780018 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:42.780001    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:15:43.231823 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:43.231789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" event={"ID":"d19f1dea-795a-4047-a006-874adb74bef4","Type":"ContainerStarted","Data":"dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6"}
Jul 31 15:15:43.231996 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:43.231840    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" event={"ID":"d19f1dea-795a-4047-a006-874adb74bef4","Type":"ContainerStarted","Data":"9ef1019f52949ca0df1fb2b2d614fea4708ea8c06db81f9ee664449c8dafeb7e"}
Jul 31 15:15:43.255709 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:43.255657    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" podStartSLOduration=1.25563461 podStartE2EDuration="1.25563461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:15:43.254752324 +0000 UTC m=+3782.127315586" watchObservedRunningTime="2026-07-31 15:15:43.25563461 +0000 UTC m=+3782.128197864"
Jul 31 15:15:43.627962 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:43.627930    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:43.633051 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:43.632995    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:44.237890 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:44.237858    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:44.238831 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:44.238809    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:15:45.903239 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:15:45.903206    2558 scope.go:117] "RemoveContainer" containerID="c23f9176909439a9dbb4bea348fd60aa887ccb4bd56ac55274de431b46645d4e"
Jul 31 15:16:00.280238 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:00.280191    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 31 15:16:00.280662 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:00.280265    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 31 15:16:01.343618 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:01.343586    2558 generic.go:356] "Generic (PLEG): container finished" podID="d19f1dea-795a-4047-a006-874adb74bef4" containerID="dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6" exitCode=0
Jul 31 15:16:01.344151 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:01.343656    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" event={"ID":"d19f1dea-795a-4047-a006-874adb74bef4","Type":"ContainerDied","Data":"dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6"}
Jul 31 15:16:01.344223 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:01.344176    2558 scope.go:117] "RemoveContainer" containerID="dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6"
Jul 31 15:16:01.345310 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:01.345281    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:16:01.345393 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:01.345353    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:16:01.345393 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:01.345381    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 31 15:16:02.352982 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:02.352946    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" event={"ID":"d19f1dea-795a-4047-a006-874adb74bef4","Type":"ContainerStarted","Data":"c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639"}
Jul 31 15:16:02.628812 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:02.628712    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:16:02.628995 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:02.628861    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 31 15:16:02.628995 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:02.628919    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 31 15:16:03.628752 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:03.628717    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 31 15:16:03.629396 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:03.629361    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 31 15:16:04.628420 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:04.628375    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 31 15:16:04.628645 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:04.628450    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 31 15:16:05.629109 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:05.629070    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 31 15:16:05.629612 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:05.629154    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 31 15:16:06.628647 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:06.628610    2558 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Jul 31 15:16:06.628990 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:06.628961    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Jul 31 15:16:07.632257 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:07.632226    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:16:07.632663 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:07.632641    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:16:07.633253 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:07.633233    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:16:10.895844 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:10.895812    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"]
Jul 31 15:16:10.898577 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:10.898561    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"
Jul 31 15:16:10.901095 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:10.901073    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:16:10.907693 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:10.906346    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"]
Jul 31 15:16:10.939200 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:10.939164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6df6ca-0b4d-4633-b048-ae9069fccc04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4\" (UID: \"4b6df6ca-0b4d-4633-b048-ae9069fccc04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"
Jul 31 15:16:11.039797 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:11.039747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6df6ca-0b4d-4633-b048-ae9069fccc04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4\" (UID: \"4b6df6ca-0b4d-4633-b048-ae9069fccc04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"
Jul 31 15:16:11.040447 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:11.040428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6df6ca-0b4d-4633-b048-ae9069fccc04-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4\" (UID: \"4b6df6ca-0b4d-4633-b048-ae9069fccc04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"
Jul 31 15:16:11.211001 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:11.210909    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"
Jul 31 15:16:11.351804 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:11.351775    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"]
Jul 31 15:16:11.354789 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:16:11.354763    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6df6ca_0b4d_4633_b048_ae9069fccc04.slice/crio-5b2f0808c249bc4e1c07380e225dda69700c57aec9426bca8195b2cf94d37e2d WatchSource:0}: Error finding container 5b2f0808c249bc4e1c07380e225dda69700c57aec9426bca8195b2cf94d37e2d: Status 404 returned error can't find the container with id 5b2f0808c249bc4e1c07380e225dda69700c57aec9426bca8195b2cf94d37e2d
Jul 31 15:16:11.407181 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:11.407151    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4" event={"ID":"4b6df6ca-0b4d-4633-b048-ae9069fccc04","Type":"ContainerStarted","Data":"5b2f0808c249bc4e1c07380e225dda69700c57aec9426bca8195b2cf94d37e2d"}
Jul 31 15:16:12.414034 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:12.413985    2558 generic.go:356] "Generic (PLEG): container finished" podID="4b6df6ca-0b4d-4633-b048-ae9069fccc04" containerID="2ff36969716b4b7bb6d976f2c0870a73f37bd0be47b7cdb5d009f59fa6c65ce1" exitCode=0
Jul 31 15:16:12.414460 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:12.414070    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4" event={"ID":"4b6df6ca-0b4d-4633-b048-ae9069fccc04","Type":"ContainerDied","Data":"2ff36969716b4b7bb6d976f2c0870a73f37bd0be47b7cdb5d009f59fa6c65ce1"}
Jul 31 15:16:13.549777 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:13.549753    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"
Jul 31 15:16:13.663986 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:13.663955    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6df6ca-0b4d-4633-b048-ae9069fccc04-maas-api-service-ca\") pod \"4b6df6ca-0b4d-4633-b048-ae9069fccc04\" (UID: \"4b6df6ca-0b4d-4633-b048-ae9069fccc04\") "
Jul 31 15:16:13.664357 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:13.664330    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6df6ca-0b4d-4633-b048-ae9069fccc04-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b6df6ca-0b4d-4633-b048-ae9069fccc04" (UID: "4b6df6ca-0b4d-4633-b048-ae9069fccc04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:16:13.765171 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:13.765097    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6df6ca-0b4d-4633-b048-ae9069fccc04-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:14.427141 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:14.427105    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"
Jul 31 15:16:14.427321 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:14.427151    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4" event={"ID":"4b6df6ca-0b4d-4633-b048-ae9069fccc04","Type":"ContainerDied","Data":"5b2f0808c249bc4e1c07380e225dda69700c57aec9426bca8195b2cf94d37e2d"}
Jul 31 15:16:14.427321 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:14.427202    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b2f0808c249bc4e1c07380e225dda69700c57aec9426bca8195b2cf94d37e2d"
Jul 31 15:16:16.965408 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:16.965371    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"]
Jul 31 15:16:16.970197 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:16.970167    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-39573e-79qg4"]
Jul 31 15:16:17.579986 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:17.579950    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6df6ca-0b4d-4633-b048-ae9069fccc04" path="/var/lib/kubelet/pods/4b6df6ca-0b4d-4633-b048-ae9069fccc04/volumes"
Jul 31 15:16:23.455339 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:23.455304    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"]
Jul 31 15:16:23.455761 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:23.455549    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy" containerID="cri-o://c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639" gracePeriod=30
Jul 31 15:16:27.226841 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.226805    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"]
Jul 31 15:16:27.227278 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.227262    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6df6ca-0b4d-4633-b048-ae9069fccc04" containerName="revoke-keys"
Jul 31 15:16:27.227325 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.227280    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6df6ca-0b4d-4633-b048-ae9069fccc04" containerName="revoke-keys"
Jul 31 15:16:27.227358 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.227351    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6df6ca-0b4d-4633-b048-ae9069fccc04" containerName="revoke-keys"
Jul 31 15:16:27.229420 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.229404    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"
Jul 31 15:16:27.232155 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.232135    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:16:27.238440 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.238416    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"]
Jul 31 15:16:27.302576 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.302540    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa0bf1-b2c6-4fbd-a435-6b6941cd179a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58af04-55drh\" (UID: \"12fa0bf1-b2c6-4fbd-a435-6b6941cd179a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"
Jul 31 15:16:27.403632 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.403599    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa0bf1-b2c6-4fbd-a435-6b6941cd179a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58af04-55drh\" (UID: \"12fa0bf1-b2c6-4fbd-a435-6b6941cd179a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"
Jul 31 15:16:27.406017 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.404266    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa0bf1-b2c6-4fbd-a435-6b6941cd179a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58af04-55drh\" (UID: \"12fa0bf1-b2c6-4fbd-a435-6b6941cd179a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"
Jul 31 15:16:27.541723 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.541692    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"
Jul 31 15:16:27.679256 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:27.679227    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"]
Jul 31 15:16:27.680269 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:16:27.680236    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fa0bf1_b2c6_4fbd_a435_6b6941cd179a.slice/crio-092b54a4ea47726a20c995c701c483d146bc7adc1dcc03eba3ae9d76d61e136d WatchSource:0}: Error finding container 092b54a4ea47726a20c995c701c483d146bc7adc1dcc03eba3ae9d76d61e136d: Status 404 returned error can't find the container with id 092b54a4ea47726a20c995c701c483d146bc7adc1dcc03eba3ae9d76d61e136d
Jul 31 15:16:28.525499 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.525470    2558 generic.go:356] "Generic (PLEG): container finished" podID="12fa0bf1-b2c6-4fbd-a435-6b6941cd179a" containerID="12dc3b9c4e1a9d6903f38344502bc9af925073bcc6f08d9bea47a66aa16cbc72" exitCode=0
Jul 31 15:16:28.525864 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.525530    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh" event={"ID":"12fa0bf1-b2c6-4fbd-a435-6b6941cd179a","Type":"ContainerDied","Data":"12dc3b9c4e1a9d6903f38344502bc9af925073bcc6f08d9bea47a66aa16cbc72"}
Jul 31 15:16:28.525864 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.525574    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh" event={"ID":"12fa0bf1-b2c6-4fbd-a435-6b6941cd179a","Type":"ContainerStarted","Data":"092b54a4ea47726a20c995c701c483d146bc7adc1dcc03eba3ae9d76d61e136d"}
Jul 31 15:16:28.710747 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.710720    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:16:28.819824 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.819791    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19f1dea-795a-4047-a006-874adb74bef4-istiod-ca-cert\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.819824 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.819837    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-socket\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.820110 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.819865    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-credential-socket\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.820110 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.819984    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-envoy\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.820110 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820088    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19f1dea-795a-4047-a006-874adb74bef4-istio-podinfo\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.820276 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820133    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-istio-token\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.820276 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:28.820276 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820177    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-data\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.820276 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820241    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:28.820496 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820291    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-certs\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.820496 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820324    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snt8x\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-kube-api-access-snt8x\") pod \"d19f1dea-795a-4047-a006-874adb74bef4\" (UID: \"d19f1dea-795a-4047-a006-874adb74bef4\") "
Jul 31 15:16:28.820496 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820367    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-data" (OuterVolumeSpecName: "istio-data") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:28.820496 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820403    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19f1dea-795a-4047-a006-874adb74bef4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:16:28.820683 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820560    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:28.820776 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820754    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:28.820853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820782    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:28.820853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820796    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d19f1dea-795a-4047-a006-874adb74bef4-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:28.820853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820808    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:28.820853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.820822    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:28.822331 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.822306    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-istio-token" (OuterVolumeSpecName: "istio-token") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:16:28.822430 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.822311    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:16:28.822430 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.822327    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d19f1dea-795a-4047-a006-874adb74bef4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 15:16:28.822430 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.822383    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-kube-api-access-snt8x" (OuterVolumeSpecName: "kube-api-access-snt8x") pod "d19f1dea-795a-4047-a006-874adb74bef4" (UID: "d19f1dea-795a-4047-a006-874adb74bef4"). InnerVolumeSpecName "kube-api-access-snt8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:16:28.921519 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.921490    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d19f1dea-795a-4047-a006-874adb74bef4-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:28.921519 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.921514    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d19f1dea-795a-4047-a006-874adb74bef4-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:28.921519 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.921526    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:28.921737 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:28.921534    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snt8x\" (UniqueName: \"kubernetes.io/projected/d19f1dea-795a-4047-a006-874adb74bef4-kube-api-access-snt8x\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:29.532539 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.532506    2558 generic.go:356] "Generic (PLEG): container finished" podID="d19f1dea-795a-4047-a006-874adb74bef4" containerID="c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639" exitCode=0
Jul 31 15:16:29.532951 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.532551    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"
Jul 31 15:16:29.532951 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.532581    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" event={"ID":"d19f1dea-795a-4047-a006-874adb74bef4","Type":"ContainerDied","Data":"c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639"}
Jul 31 15:16:29.532951 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.532636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5" event={"ID":"d19f1dea-795a-4047-a006-874adb74bef4","Type":"ContainerDied","Data":"9ef1019f52949ca0df1fb2b2d614fea4708ea8c06db81f9ee664449c8dafeb7e"}
Jul 31 15:16:29.532951 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.532658    2558 scope.go:117] "RemoveContainer" containerID="c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639"
Jul 31 15:16:29.543136 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.543117    2558 scope.go:117] "RemoveContainer" containerID="dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6"
Jul 31 15:16:29.552419 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.552399    2558 scope.go:117] "RemoveContainer" containerID="c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639"
Jul 31 15:16:29.552794 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:16:29.552754    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639\": container with ID starting with c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639 not found: ID does not exist" containerID="c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639"
Jul 31 15:16:29.552899 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.552807    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639"} err="failed to get container status \"c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639\": rpc error: code = NotFound desc = could not find container \"c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639\": container with ID starting with c8819fcdd92188624011c5ace8c6f06452101a19b0c9750e9126161522b75639 not found: ID does not exist"
Jul 31 15:16:29.552899 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.552834    2558 scope.go:117] "RemoveContainer" containerID="dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6"
Jul 31 15:16:29.553201 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:16:29.553166    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6\": container with ID starting with dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6 not found: ID does not exist" containerID="dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6"
Jul 31 15:16:29.553297 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.553205    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6"} err="failed to get container status \"dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6\": rpc error: code = NotFound desc = could not find container \"dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6\": container with ID starting with dc52a93fcf68179a3e7270af40d530264c118ae9d65101615c9a4454f56b67c6 not found: ID does not exist"
Jul 31 15:16:29.556569 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.556545    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"]
Jul 31 15:16:29.562050 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.562016    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-39573e-openshift-default-5b594664b8-n8rf5"]
Jul 31 15:16:29.580137 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.580114    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d19f1dea-795a-4047-a006-874adb74bef4" path="/var/lib/kubelet/pods/d19f1dea-795a-4047-a006-874adb74bef4/volumes"
Jul 31 15:16:29.674787 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.674764    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"
Jul 31 15:16:29.831827 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.831790    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa0bf1-b2c6-4fbd-a435-6b6941cd179a-maas-api-service-ca\") pod \"12fa0bf1-b2c6-4fbd-a435-6b6941cd179a\" (UID: \"12fa0bf1-b2c6-4fbd-a435-6b6941cd179a\") "
Jul 31 15:16:29.832175 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.832153    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12fa0bf1-b2c6-4fbd-a435-6b6941cd179a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12fa0bf1-b2c6-4fbd-a435-6b6941cd179a" (UID: "12fa0bf1-b2c6-4fbd-a435-6b6941cd179a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:16:29.933527 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:29.933488    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12fa0bf1-b2c6-4fbd-a435-6b6941cd179a-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:16:30.540731 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:30.540703    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"
Jul 31 15:16:30.540731 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:30.540718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh" event={"ID":"12fa0bf1-b2c6-4fbd-a435-6b6941cd179a","Type":"ContainerDied","Data":"092b54a4ea47726a20c995c701c483d146bc7adc1dcc03eba3ae9d76d61e136d"}
Jul 31 15:16:30.541290 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:30.540764    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="092b54a4ea47726a20c995c701c483d146bc7adc1dcc03eba3ae9d76d61e136d"
Jul 31 15:16:42.678278 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:42.678242    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"]
Jul 31 15:16:42.680515 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:42.680494    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-58af04-29758515-hf8mt"]
Jul 31 15:16:42.826440 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:42.826399    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"]
Jul 31 15:16:42.829851 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:42.829822    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58af04-55drh"]
Jul 31 15:16:43.580390 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:43.580357    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12fa0bf1-b2c6-4fbd-a435-6b6941cd179a" path="/var/lib/kubelet/pods/12fa0bf1-b2c6-4fbd-a435-6b6941cd179a/volumes"
Jul 31 15:16:43.580686 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:43.580674    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="594bdf05-b917-42c1-87bb-6e8995c0596c" path="/var/lib/kubelet/pods/594bdf05-b917-42c1-87bb-6e8995c0596c/volumes"
Jul 31 15:16:48.197007 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:16:48.196973    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"]
Jul 31 15:17:03.217016 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.216946    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9" podUID="f4abbd14-9fa4-44c3-9383-3eda3cfd6718" containerName="main" containerID="cri-o://d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97" gracePeriod=45
Jul 31 15:17:03.466263 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.466239    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:17:03.567048 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567005    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-model-cache\") pod \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") "
Jul 31 15:17:03.567239 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567079    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-dshm\") pod \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") "
Jul 31 15:17:03.567239 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567121    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-home\") pod \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") "
Jul 31 15:17:03.567239 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567181    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tmp-dir\") pod \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") "
Jul 31 15:17:03.567239 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45hvb\" (UniqueName: \"kubernetes.io/projected/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-kube-api-access-45hvb\") pod \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") "
Jul 31 15:17:03.567459 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567234    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4abbd14-9fa4-44c3-9383-3eda3cfd6718" (UID: "f4abbd14-9fa4-44c3-9383-3eda3cfd6718"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:03.567459 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567300    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tls-certs\") pod \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\" (UID: \"f4abbd14-9fa4-44c3-9383-3eda3cfd6718\") "
Jul 31 15:17:03.567459 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567440    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4abbd14-9fa4-44c3-9383-3eda3cfd6718" (UID: "f4abbd14-9fa4-44c3-9383-3eda3cfd6718"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:03.567579 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567454    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-home" (OuterVolumeSpecName: "home") pod "f4abbd14-9fa4-44c3-9383-3eda3cfd6718" (UID: "f4abbd14-9fa4-44c3-9383-3eda3cfd6718"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:03.567684 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567653    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-home\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:03.567684 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567683    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tmp-dir\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:03.567873 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.567692    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-model-cache\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:03.569476 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.569452    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-dshm" (OuterVolumeSpecName: "dshm") pod "f4abbd14-9fa4-44c3-9383-3eda3cfd6718" (UID: "f4abbd14-9fa4-44c3-9383-3eda3cfd6718"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:03.569576 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.569507    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-kube-api-access-45hvb" (OuterVolumeSpecName: "kube-api-access-45hvb") pod "f4abbd14-9fa4-44c3-9383-3eda3cfd6718" (UID: "f4abbd14-9fa4-44c3-9383-3eda3cfd6718"). InnerVolumeSpecName "kube-api-access-45hvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:17:03.569623 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.569584    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4abbd14-9fa4-44c3-9383-3eda3cfd6718" (UID: "f4abbd14-9fa4-44c3-9383-3eda3cfd6718"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:17:03.668787 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.668758    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-dshm\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:03.668787 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.668784    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45hvb\" (UniqueName: \"kubernetes.io/projected/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-kube-api-access-45hvb\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:03.668787 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.668794    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4abbd14-9fa4-44c3-9383-3eda3cfd6718-tls-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:03.738960 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.738927    2558 generic.go:356] "Generic (PLEG): container finished" podID="f4abbd14-9fa4-44c3-9383-3eda3cfd6718" containerID="d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97" exitCode=0
Jul 31 15:17:03.739142 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.738976    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"
Jul 31 15:17:03.739142 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.739014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9" event={"ID":"f4abbd14-9fa4-44c3-9383-3eda3cfd6718","Type":"ContainerDied","Data":"d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97"}
Jul 31 15:17:03.739142 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.739088    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9" event={"ID":"f4abbd14-9fa4-44c3-9383-3eda3cfd6718","Type":"ContainerDied","Data":"45fe2ca612639252f8061f604b1952e5fdde2d75b8c65ec7eff861274537bc8a"}
Jul 31 15:17:03.739142 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.739114    2558 scope.go:117] "RemoveContainer" containerID="d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97"
Jul 31 15:17:03.748434 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.748416    2558 scope.go:117] "RemoveContainer" containerID="d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97"
Jul 31 15:17:03.748720 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:17:03.748701    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97\": container with ID starting with d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97 not found: ID does not exist" containerID="d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97"
Jul 31 15:17:03.748799 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.748729    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97"} err="failed to get container status \"d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97\": rpc error: code = NotFound desc = could not find container \"d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97\": container with ID starting with d67c2f1be347c46e7facf28093fedbbae33eff9b08cc1f9a99c107b1a0291e97 not found: ID does not exist"
Jul 31 15:17:03.762905 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.762872    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"]
Jul 31 15:17:03.763064 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:03.762924    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-58af04/ipp-route-58af04-kserve-68ff6dfd79-652h9"]
Jul 31 15:17:05.580404 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:05.580368    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4abbd14-9fa4-44c3-9383-3eda3cfd6718" path="/var/lib/kubelet/pods/f4abbd14-9fa4-44c3-9383-3eda3cfd6718/volumes"
Jul 31 15:17:10.992843 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.992810    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"]
Jul 31 15:17:10.993489 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993465    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy"
Jul 31 15:17:10.993603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993491    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy"
Jul 31 15:17:10.993603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993502    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12fa0bf1-b2c6-4fbd-a435-6b6941cd179a" containerName="revoke-keys"
Jul 31 15:17:10.993603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993513    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="12fa0bf1-b2c6-4fbd-a435-6b6941cd179a" containerName="revoke-keys"
Jul 31 15:17:10.993603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993525    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy"
Jul 31 15:17:10.993603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993534    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy"
Jul 31 15:17:10.993603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993550    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4abbd14-9fa4-44c3-9383-3eda3cfd6718" containerName="main"
Jul 31 15:17:10.993603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993558    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4abbd14-9fa4-44c3-9383-3eda3cfd6718" containerName="main"
Jul 31 15:17:10.993952 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993704    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="12fa0bf1-b2c6-4fbd-a435-6b6941cd179a" containerName="revoke-keys"
Jul 31 15:17:10.993952 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993716    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy"
Jul 31 15:17:10.993952 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993727    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4abbd14-9fa4-44c3-9383-3eda3cfd6718" containerName="main"
Jul 31 15:17:10.993952 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.993740    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d19f1dea-795a-4047-a006-874adb74bef4" containerName="istio-proxy"
Jul 31 15:17:10.998386 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:10.998366    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"
Jul 31 15:17:11.002123 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.002089    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:17:11.004828 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.004795    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"]
Jul 31 15:17:11.140756 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.140724    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32c6c38-1632-4e8a-9af6-3fee4ca7202d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj\" (UID: \"f32c6c38-1632-4e8a-9af6-3fee4ca7202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"
Jul 31 15:17:11.242158 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.242114    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32c6c38-1632-4e8a-9af6-3fee4ca7202d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj\" (UID: \"f32c6c38-1632-4e8a-9af6-3fee4ca7202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"
Jul 31 15:17:11.242798 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.242776    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32c6c38-1632-4e8a-9af6-3fee4ca7202d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj\" (UID: \"f32c6c38-1632-4e8a-9af6-3fee4ca7202d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"
Jul 31 15:17:11.311232 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.311206    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"
Jul 31 15:17:11.454432 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.454404    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"]
Jul 31 15:17:11.457056 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:17:11.456989    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32c6c38_1632_4e8a_9af6_3fee4ca7202d.slice/crio-d68ecc2b4b31499dbc7d0bec63853aced32c2aa0fa78c4a0af1a2f3f3cd6d8fa WatchSource:0}: Error finding container d68ecc2b4b31499dbc7d0bec63853aced32c2aa0fa78c4a0af1a2f3f3cd6d8fa: Status 404 returned error can't find the container with id d68ecc2b4b31499dbc7d0bec63853aced32c2aa0fa78c4a0af1a2f3f3cd6d8fa
Jul 31 15:17:11.788692 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.788652    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj" event={"ID":"f32c6c38-1632-4e8a-9af6-3fee4ca7202d","Type":"ContainerStarted","Data":"053c454f902d74a0de2862ce689b640c8f46bfe82e0501c208f0f48ce7421df5"}
Jul 31 15:17:11.788845 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.788708    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj" event={"ID":"f32c6c38-1632-4e8a-9af6-3fee4ca7202d","Type":"ContainerStarted","Data":"d68ecc2b4b31499dbc7d0bec63853aced32c2aa0fa78c4a0af1a2f3f3cd6d8fa"}
Jul 31 15:17:11.807733 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:11.807685    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj" podStartSLOduration=1.80767078 podStartE2EDuration="1.80767078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:11.807347032 +0000 UTC m=+3870.679910310" watchObservedRunningTime="2026-07-31 15:17:11.80767078 +0000 UTC m=+3870.680234014"
Jul 31 15:17:12.796077 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:12.796010    2558 generic.go:356] "Generic (PLEG): container finished" podID="f32c6c38-1632-4e8a-9af6-3fee4ca7202d" containerID="053c454f902d74a0de2862ce689b640c8f46bfe82e0501c208f0f48ce7421df5" exitCode=0
Jul 31 15:17:12.796577 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:12.796096    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj" event={"ID":"f32c6c38-1632-4e8a-9af6-3fee4ca7202d","Type":"ContainerDied","Data":"053c454f902d74a0de2862ce689b640c8f46bfe82e0501c208f0f48ce7421df5"}
Jul 31 15:17:13.945078 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:13.945056    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"
Jul 31 15:17:13.963340 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:13.963313    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32c6c38-1632-4e8a-9af6-3fee4ca7202d-maas-api-service-ca\") pod \"f32c6c38-1632-4e8a-9af6-3fee4ca7202d\" (UID: \"f32c6c38-1632-4e8a-9af6-3fee4ca7202d\") "
Jul 31 15:17:13.963716 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:13.963690    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32c6c38-1632-4e8a-9af6-3fee4ca7202d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f32c6c38-1632-4e8a-9af6-3fee4ca7202d" (UID: "f32c6c38-1632-4e8a-9af6-3fee4ca7202d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:14.064776 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:14.064681    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32c6c38-1632-4e8a-9af6-3fee4ca7202d-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:14.808985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:14.808951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj" event={"ID":"f32c6c38-1632-4e8a-9af6-3fee4ca7202d","Type":"ContainerDied","Data":"d68ecc2b4b31499dbc7d0bec63853aced32c2aa0fa78c4a0af1a2f3f3cd6d8fa"}
Jul 31 15:17:14.808985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:14.808975    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"
Jul 31 15:17:14.809218 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:14.809000    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d68ecc2b4b31499dbc7d0bec63853aced32c2aa0fa78c4a0af1a2f3f3cd6d8fa"
Jul 31 15:17:21.380960 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:21.380926    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"]
Jul 31 15:17:21.383768 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:21.383743    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1c9d21-29758515-m9r8s"]
Jul 31 15:17:21.537431 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:21.537390    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"]
Jul 31 15:17:21.539548 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:21.539529    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1c9d21-sw8nj"]
Jul 31 15:17:21.584302 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:21.584262    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e21f4200-a559-4dd2-9711-90bbd84bfdb3" path="/var/lib/kubelet/pods/e21f4200-a559-4dd2-9711-90bbd84bfdb3/volumes"
Jul 31 15:17:21.584761 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:21.584739    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f32c6c38-1632-4e8a-9af6-3fee4ca7202d" path="/var/lib/kubelet/pods/f32c6c38-1632-4e8a-9af6-3fee4ca7202d/volumes"
Jul 31 15:17:28.202460 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.202425    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"]
Jul 31 15:17:28.202933 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.202705    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy" containerID="cri-o://c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d" gracePeriod=30
Jul 31 15:17:28.623766 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.623731    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"]
Jul 31 15:17:28.624229 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.624216    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f32c6c38-1632-4e8a-9af6-3fee4ca7202d" containerName="revoke-keys"
Jul 31 15:17:28.624229 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.624230    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f32c6c38-1632-4e8a-9af6-3fee4ca7202d" containerName="revoke-keys"
Jul 31 15:17:28.624372 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.624308    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f32c6c38-1632-4e8a-9af6-3fee4ca7202d" containerName="revoke-keys"
Jul 31 15:17:28.627507 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.627486    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"
Jul 31 15:17:28.630511 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.630488    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:17:28.637193 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.637163    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"]
Jul 31 15:17:28.708805 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.708777    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e656469e-ee58-4077-9378-22ba7cc753a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9966b2-9447h\" (UID: \"e656469e-ee58-4077-9378-22ba7cc753a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"
Jul 31 15:17:28.809572 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.809536    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e656469e-ee58-4077-9378-22ba7cc753a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9966b2-9447h\" (UID: \"e656469e-ee58-4077-9378-22ba7cc753a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"
Jul 31 15:17:28.810168 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.810152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e656469e-ee58-4077-9378-22ba7cc753a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-9966b2-9447h\" (UID: \"e656469e-ee58-4077-9378-22ba7cc753a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"
Jul 31 15:17:28.939783 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:28.939702    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"
Jul 31 15:17:29.081608 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:29.081580    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"]
Jul 31 15:17:29.083253 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:17:29.083225    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode656469e_ee58_4077_9378_22ba7cc753a9.slice/crio-548594b537bdce79e644f0497fc43791ee75a63fcb3f56973b9003949cccc71c WatchSource:0}: Error finding container 548594b537bdce79e644f0497fc43791ee75a63fcb3f56973b9003949cccc71c: Status 404 returned error can't find the container with id 548594b537bdce79e644f0497fc43791ee75a63fcb3f56973b9003949cccc71c
Jul 31 15:17:29.902763 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:29.902730    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h" event={"ID":"e656469e-ee58-4077-9378-22ba7cc753a9","Type":"ContainerStarted","Data":"089f39978472770513a7ef648389b4077a4a6edaf49632f8d521f892a7e42a9a"}
Jul 31 15:17:29.903165 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:29.902782    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h" event={"ID":"e656469e-ee58-4077-9378-22ba7cc753a9","Type":"ContainerStarted","Data":"548594b537bdce79e644f0497fc43791ee75a63fcb3f56973b9003949cccc71c"}
Jul 31 15:17:29.934387 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:29.934341    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h" podStartSLOduration=1.934324465 podStartE2EDuration="1.934324465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:17:29.931171181 +0000 UTC m=+3888.803734435" watchObservedRunningTime="2026-07-31 15:17:29.934324465 +0000 UTC m=+3888.806887732"
Jul 31 15:17:30.911061 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:30.911015    2558 generic.go:356] "Generic (PLEG): container finished" podID="e656469e-ee58-4077-9378-22ba7cc753a9" containerID="089f39978472770513a7ef648389b4077a4a6edaf49632f8d521f892a7e42a9a" exitCode=0
Jul 31 15:17:30.911427 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:30.911115    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h" event={"ID":"e656469e-ee58-4077-9378-22ba7cc753a9","Type":"ContainerDied","Data":"089f39978472770513a7ef648389b4077a4a6edaf49632f8d521f892a7e42a9a"}
Jul 31 15:17:32.062828 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:32.062792    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"
Jul 31 15:17:32.143243 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:32.143210    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e656469e-ee58-4077-9378-22ba7cc753a9-maas-api-service-ca\") pod \"e656469e-ee58-4077-9378-22ba7cc753a9\" (UID: \"e656469e-ee58-4077-9378-22ba7cc753a9\") "
Jul 31 15:17:32.143545 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:32.143523    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e656469e-ee58-4077-9378-22ba7cc753a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e656469e-ee58-4077-9378-22ba7cc753a9" (UID: "e656469e-ee58-4077-9378-22ba7cc753a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:32.244515 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:32.244436    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e656469e-ee58-4077-9378-22ba7cc753a9-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:32.923755 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:32.923725    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"
Jul 31 15:17:32.923932 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:32.923773    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h" event={"ID":"e656469e-ee58-4077-9378-22ba7cc753a9","Type":"ContainerDied","Data":"548594b537bdce79e644f0497fc43791ee75a63fcb3f56973b9003949cccc71c"}
Jul 31 15:17:32.923932 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:32.923819    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="548594b537bdce79e644f0497fc43791ee75a63fcb3f56973b9003949cccc71c"
Jul 31 15:17:33.459536 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.459512    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:17:33.556956 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.556921    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-certs\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557164 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.556980    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-data\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557164 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557018    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-credential-socket\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557164 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557060    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm9fl\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-kube-api-access-pm9fl\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557164 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557079    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-podinfo\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557164 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557106    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-token\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557443 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557244    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istiod-ca-cert\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557443 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557272    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:33.557443 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557309    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-socket\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557443 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-envoy\") pod \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\" (UID: \"bcf695c4-4163-4e4d-9bd9-7180543b9e1a\") "
Jul 31 15:17:33.557656 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557553    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:33.557716 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557667    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-data" (OuterVolumeSpecName: "istio-data") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:33.557716 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557684    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:33.557820 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557729    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:33.557820 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557802    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-certs\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.557912 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557823    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-data\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.557912 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557837    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-credential-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.557912 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.557852    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-workload-socket\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.559367 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.559329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 31 15:17:33.559491 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.559407    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:33.559619 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.559603    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-kube-api-access-pm9fl" (OuterVolumeSpecName: "kube-api-access-pm9fl") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "kube-api-access-pm9fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:17:33.559740 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.559718    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-token" (OuterVolumeSpecName: "istio-token") pod "bcf695c4-4163-4e4d-9bd9-7180543b9e1a" (UID: "bcf695c4-4163-4e4d-9bd9-7180543b9e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:17:33.658884 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.658857    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9fl\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-kube-api-access-pm9fl\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.658884 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.658883    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-podinfo\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.659094 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.658893    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-token\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.659094 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.658901    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istiod-ca-cert\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.659094 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.658908    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcf695c4-4163-4e4d-9bd9-7180543b9e1a-istio-envoy\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:33.931890 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.931794    2558 generic.go:356] "Generic (PLEG): container finished" podID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerID="c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d" exitCode=0
Jul 31 15:17:33.931890 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.931845    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"
Jul 31 15:17:33.932090 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.931881    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" event={"ID":"bcf695c4-4163-4e4d-9bd9-7180543b9e1a","Type":"ContainerDied","Data":"c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d"}
Jul 31 15:17:33.932090 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.931951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc" event={"ID":"bcf695c4-4163-4e4d-9bd9-7180543b9e1a","Type":"ContainerDied","Data":"6f63472579ee6d82f8755ab41a510cd788082e746ac742b9380b17dd5e3372b9"}
Jul 31 15:17:33.932090 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.931981    2558 scope.go:117] "RemoveContainer" containerID="c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d"
Jul 31 15:17:33.941566 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.941548    2558 scope.go:117] "RemoveContainer" containerID="34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4"
Jul 31 15:17:33.950449 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.950430    2558 scope.go:117] "RemoveContainer" containerID="c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d"
Jul 31 15:17:33.950716 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:17:33.950693    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d\": container with ID starting with c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d not found: ID does not exist" containerID="c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d"
Jul 31 15:17:33.950784 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.950727    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d"} err="failed to get container status \"c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d\": rpc error: code = NotFound desc = could not find container \"c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d\": container with ID starting with c3e17529935935b194b15620650ce309f2905a4c7c7994705116e04b3262039d not found: ID does not exist"
Jul 31 15:17:33.950784 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.950750    2558 scope.go:117] "RemoveContainer" containerID="34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4"
Jul 31 15:17:33.950894 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.950787    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"]
Jul 31 15:17:33.951054 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:17:33.951014    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4\": container with ID starting with 34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4 not found: ID does not exist" containerID="34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4"
Jul 31 15:17:33.951106 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.951062    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4"} err="failed to get container status \"34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4\": rpc error: code = NotFound desc = could not find container \"34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4\": container with ID starting with 34815c28dae284f80daa6ead81798f25bfee6c3f49175a822fd616a63c27b4b4 not found: ID does not exist"
Jul 31 15:17:33.957216 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:33.957196    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c9d21-openshift-default-5468968b5b-cdvgc"]
Jul 31 15:17:35.581477 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:35.581444    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" path="/var/lib/kubelet/pods/bcf695c4-4163-4e4d-9bd9-7180543b9e1a/volumes"
Jul 31 15:17:39.149523 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:39.149467    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"]
Jul 31 15:17:39.155862 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:39.155832    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-9966b2-29758515-xqlzz"]
Jul 31 15:17:39.319194 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:39.319093    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"]
Jul 31 15:17:39.323500 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:39.323468    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-9966b2-9447h"]
Jul 31 15:17:39.580300 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:39.580266    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e656469e-ee58-4077-9378-22ba7cc753a9" path="/var/lib/kubelet/pods/e656469e-ee58-4077-9378-22ba7cc753a9/volumes"
Jul 31 15:17:39.580635 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:39.580614    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef2c8b54-846a-4b59-b5dd-0f806ac5a810" path="/var/lib/kubelet/pods/ef2c8b54-846a-4b59-b5dd-0f806ac5a810/volumes"
Jul 31 15:17:45.688841 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:45.688802    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:17:45.698626 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:45.698605    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:17:46.272635 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.272550    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"]
Jul 31 15:17:46.273044 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273016    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy"
Jul 31 15:17:46.273098 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273046    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy"
Jul 31 15:17:46.273098 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273055    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy"
Jul 31 15:17:46.273098 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273061    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy"
Jul 31 15:17:46.273098 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273079    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e656469e-ee58-4077-9378-22ba7cc753a9" containerName="revoke-keys"
Jul 31 15:17:46.273098 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273086    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e656469e-ee58-4077-9378-22ba7cc753a9" containerName="revoke-keys"
Jul 31 15:17:46.273256 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273155    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e656469e-ee58-4077-9378-22ba7cc753a9" containerName="revoke-keys"
Jul 31 15:17:46.273256 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273166    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy"
Jul 31 15:17:46.273256 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.273173    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcf695c4-4163-4e4d-9bd9-7180543b9e1a" containerName="istio-proxy"
Jul 31 15:17:46.277935 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.277918    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"
Jul 31 15:17:46.280524 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.280497    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lb2sd\""
Jul 31 15:17:46.283961 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.283939    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"]
Jul 31 15:17:46.376697 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.376665    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0398f0-adf3-4ff4-96a7-1d67619c9318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt\" (UID: \"3d0398f0-adf3-4ff4-96a7-1d67619c9318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"
Jul 31 15:17:46.477907 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.477878    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0398f0-adf3-4ff4-96a7-1d67619c9318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt\" (UID: \"3d0398f0-adf3-4ff4-96a7-1d67619c9318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"
Jul 31 15:17:46.478574 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.478555    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0398f0-adf3-4ff4-96a7-1d67619c9318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt\" (UID: \"3d0398f0-adf3-4ff4-96a7-1d67619c9318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"
Jul 31 15:17:46.592280 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.592243    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"
Jul 31 15:17:46.732155 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:46.732132    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"]
Jul 31 15:17:46.734573 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:17:46.734547    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0398f0_adf3_4ff4_96a7_1d67619c9318.slice/crio-fc96c4ee1913b175eeac06d6fe7ae7f894409fb11a0ac6aa2187278870da4c00 WatchSource:0}: Error finding container fc96c4ee1913b175eeac06d6fe7ae7f894409fb11a0ac6aa2187278870da4c00: Status 404 returned error can't find the container with id fc96c4ee1913b175eeac06d6fe7ae7f894409fb11a0ac6aa2187278870da4c00
Jul 31 15:17:47.007564 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:47.007527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt" event={"ID":"3d0398f0-adf3-4ff4-96a7-1d67619c9318","Type":"ContainerStarted","Data":"fc96c4ee1913b175eeac06d6fe7ae7f894409fb11a0ac6aa2187278870da4c00"}
Jul 31 15:17:48.014732 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:48.014690    2558 generic.go:356] "Generic (PLEG): container finished" podID="3d0398f0-adf3-4ff4-96a7-1d67619c9318" containerID="1804942efe6cc5237ed9a42445ae6c97fd07928b6cb4b45b32eb4d2def7b46e5" exitCode=0
Jul 31 15:17:48.015178 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:48.014774    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt" event={"ID":"3d0398f0-adf3-4ff4-96a7-1d67619c9318","Type":"ContainerDied","Data":"1804942efe6cc5237ed9a42445ae6c97fd07928b6cb4b45b32eb4d2def7b46e5"}
Jul 31 15:17:49.155716 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:49.155693    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"
Jul 31 15:17:49.312279 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:49.312253    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0398f0-adf3-4ff4-96a7-1d67619c9318-maas-api-service-ca\") pod \"3d0398f0-adf3-4ff4-96a7-1d67619c9318\" (UID: \"3d0398f0-adf3-4ff4-96a7-1d67619c9318\") "
Jul 31 15:17:49.312677 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:49.312656    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d0398f0-adf3-4ff4-96a7-1d67619c9318-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d0398f0-adf3-4ff4-96a7-1d67619c9318" (UID: "3d0398f0-adf3-4ff4-96a7-1d67619c9318"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:49.413624 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:49.413588    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d0398f0-adf3-4ff4-96a7-1d67619c9318-maas-api-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:50.031747 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:50.031667    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"
Jul 31 15:17:50.031747 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:50.031668    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt" event={"ID":"3d0398f0-adf3-4ff4-96a7-1d67619c9318","Type":"ContainerDied","Data":"fc96c4ee1913b175eeac06d6fe7ae7f894409fb11a0ac6aa2187278870da4c00"}
Jul 31 15:17:50.031747 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:50.031733    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc96c4ee1913b175eeac06d6fe7ae7f894409fb11a0ac6aa2187278870da4c00"
Jul 31 15:17:56.625890 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.625839    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"]
Jul 31 15:17:56.627245 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.626991    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd" podUID="ba7f7208-206e-4f88-bcb3-024d48951024" containerName="maas-api" containerID="cri-o://3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637" gracePeriod=30
Jul 31 15:17:56.652695 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.652664    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"]
Jul 31 15:17:56.657363 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.657341    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-79a3ee-29758515-8txtm"]
Jul 31 15:17:56.814718 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.814653    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"]
Jul 31 15:17:56.817513 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.817481    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-79a3ee-rrjjt"]
Jul 31 15:17:56.894433 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.894411    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:17:56.985240 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.985207    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7f7208-206e-4f88-bcb3-024d48951024-maas-api-tls\") pod \"ba7f7208-206e-4f88-bcb3-024d48951024\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") "
Jul 31 15:17:56.985240 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.985246    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f7208-206e-4f88-bcb3-024d48951024-openshift-service-ca\") pod \"ba7f7208-206e-4f88-bcb3-024d48951024\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") "
Jul 31 15:17:56.985501 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.985271    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lmth\" (UniqueName: \"kubernetes.io/projected/ba7f7208-206e-4f88-bcb3-024d48951024-kube-api-access-9lmth\") pod \"ba7f7208-206e-4f88-bcb3-024d48951024\" (UID: \"ba7f7208-206e-4f88-bcb3-024d48951024\") "
Jul 31 15:17:56.985652 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.985628    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7f7208-206e-4f88-bcb3-024d48951024-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba7f7208-206e-4f88-bcb3-024d48951024" (UID: "ba7f7208-206e-4f88-bcb3-024d48951024"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:56.987433 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.987411    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7f7208-206e-4f88-bcb3-024d48951024-kube-api-access-9lmth" (OuterVolumeSpecName: "kube-api-access-9lmth") pod "ba7f7208-206e-4f88-bcb3-024d48951024" (UID: "ba7f7208-206e-4f88-bcb3-024d48951024"). InnerVolumeSpecName "kube-api-access-9lmth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:17:56.987515 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:56.987413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7f7208-206e-4f88-bcb3-024d48951024-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba7f7208-206e-4f88-bcb3-024d48951024" (UID: "ba7f7208-206e-4f88-bcb3-024d48951024"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:17:57.075557 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.075519    2558 generic.go:356] "Generic (PLEG): container finished" podID="ba7f7208-206e-4f88-bcb3-024d48951024" containerID="3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637" exitCode=0
Jul 31 15:17:57.075739 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.075569    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"
Jul 31 15:17:57.075739 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.075600    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd" event={"ID":"ba7f7208-206e-4f88-bcb3-024d48951024","Type":"ContainerDied","Data":"3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637"}
Jul 31 15:17:57.075739 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.075666    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd" event={"ID":"ba7f7208-206e-4f88-bcb3-024d48951024","Type":"ContainerDied","Data":"ddbc735504d33f4112ed9935178caa3312dbd62eb2a4e7ba68b529f8f2844698"}
Jul 31 15:17:57.075739 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.075692    2558 scope.go:117] "RemoveContainer" containerID="3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637"
Jul 31 15:17:57.085400 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.085381    2558 scope.go:117] "RemoveContainer" containerID="3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637"
Jul 31 15:17:57.085679 ip-10-0-139-143 kubenswrapper[2558]: E0731 15:17:57.085657    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637\": container with ID starting with 3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637 not found: ID does not exist" containerID="3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637"
Jul 31 15:17:57.085741 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.085692    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637"} err="failed to get container status \"3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637\": rpc error: code = NotFound desc = could not find container \"3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637\": container with ID starting with 3391f5e200613660f739c965367577c00b7037dde0cbe8a6f71fe746075c0637 not found: ID does not exist"
Jul 31 15:17:57.085895 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.085879    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba7f7208-206e-4f88-bcb3-024d48951024-maas-api-tls\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:57.085933 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.085898    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7f7208-206e-4f88-bcb3-024d48951024-openshift-service-ca\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:57.085933 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.085908    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lmth\" (UniqueName: \"kubernetes.io/projected/ba7f7208-206e-4f88-bcb3-024d48951024-kube-api-access-9lmth\") on node \"ip-10-0-139-143.ec2.internal\" DevicePath \"\""
Jul 31 15:17:57.098292 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.098267    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"]
Jul 31 15:17:57.102800 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.102780    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-79a3ee-85b455fd46-zfkgd"]
Jul 31 15:17:57.580985 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.580950    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d0398f0-adf3-4ff4-96a7-1d67619c9318" path="/var/lib/kubelet/pods/3d0398f0-adf3-4ff4-96a7-1d67619c9318/volumes"
Jul 31 15:17:57.581284 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.581269    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845c58ca-85f3-4313-88f7-8ac226431e4a" path="/var/lib/kubelet/pods/845c58ca-85f3-4313-88f7-8ac226431e4a/volumes"
Jul 31 15:17:57.581590 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:17:57.581578    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba7f7208-206e-4f88-bcb3-024d48951024" path="/var/lib/kubelet/pods/ba7f7208-206e-4f88-bcb3-024d48951024/volumes"
Jul 31 15:18:09.608935 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:09.608897    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xzlhj_54829018-37a2-4699-af96-aea18ce0a6d3/manager/0.log"
Jul 31 15:18:09.733904 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:09.733874    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l65hb_ee99d4c6-3d42-42cd-a5d2-e6a94b651804/manager/0.log"
Jul 31 15:18:09.968317 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:09.968241    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ptcph_c936181c-91ed-4e97-abc0-d63c2912a32a/server/0.log"
Jul 31 15:18:10.090711 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:10.090680    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g95vk_b42a1bc2-e21f-44db-ab62-fb98ac94c8d1/manager/0.log"
Jul 31 15:18:10.339698 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:10.339666    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-k4dzm_67a9840e-4dcf-4ed4-8e17-9026c5b17cbe/copy-manifests/0.log"
Jul 31 15:18:10.346914 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:10.346883    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-k4dzm_67a9840e-4dcf-4ed4-8e17-9026c5b17cbe/manager/0.log"
Jul 31 15:18:10.964270 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:10.964237    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29758485-ddw52_d2f97972-a6e1-4b37-8a44-35c1cc929cef/cleanup/0.log"
Jul 31 15:18:11.199098 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:11.199066    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29758515-vb5qd_6e4680bf-d449-4c90-aabc-0d2e8d07e4e4/cleanup/0.log"
Jul 31 15:18:11.320097 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:11.320066    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kv5tz_2763d69b-797f-4001-be9a-bb56210c869a/postgres/0.log"
Jul 31 15:18:12.247385 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:12.247353    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-cl6tm_0f52ba2d-1e9b-44c3-816e-b257cdbca654/manager/0.log"
Jul 31 15:18:12.363726 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:12.363677    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-vvg9h_a1852a52-681d-48a7-9e03-564888a92c53/manager/0.log"
Jul 31 15:18:12.601363 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:12.601321    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-7n8ms_b1f1fcfc-a32b-4f95-837a-e4049c8400c4/manager/0.log"
Jul 31 15:18:12.835339 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:12.835305    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-bj8kz_47dc36c0-8bcf-46cc-935d-ea4749bcc9d1/manager/0.log"
Jul 31 15:18:13.201802 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:13.201769    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c_4fa698e3-34cf-43d6-967b-492af65a098c/istio-proxy/0.log"
Jul 31 15:18:13.561813 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:13.561778    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8b44b796-zwgwr_80f5291b-3486-44ad-8db4-d9ce9301f825/kube-auth-proxy/0.log"
Jul 31 15:18:13.691059 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:13.691009    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-x79gg_7c5561a1-775e-4ccb-a096-19bbd398bde0/istio-proxy/1.log"
Jul 31 15:18:14.631478 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:14.631443    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qspdj_00856855-cb2f-4ecd-8665-bbd34dd894ab/main/0.log"
Jul 31 15:18:14.859404 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:14.859374    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-dw27j_223d2c47-0bf4-44d6-86a1-e68c18287892/main/0.log"
Jul 31 15:18:20.626357 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.626306    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7w47/must-gather-t77fq"]
Jul 31 15:18:20.626853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.626734    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d0398f0-adf3-4ff4-96a7-1d67619c9318" containerName="revoke-keys"
Jul 31 15:18:20.626853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.626745    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d0398f0-adf3-4ff4-96a7-1d67619c9318" containerName="revoke-keys"
Jul 31 15:18:20.626853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.626765    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba7f7208-206e-4f88-bcb3-024d48951024" containerName="maas-api"
Jul 31 15:18:20.626853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.626770    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba7f7208-206e-4f88-bcb3-024d48951024" containerName="maas-api"
Jul 31 15:18:20.626853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.626827    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba7f7208-206e-4f88-bcb3-024d48951024" containerName="maas-api"
Jul 31 15:18:20.626853 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.626836    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d0398f0-adf3-4ff4-96a7-1d67619c9318" containerName="revoke-keys"
Jul 31 15:18:20.630749 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.630719    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7w47/must-gather-t77fq"
Jul 31 15:18:20.633739 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.633714    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7w47\"/\"kube-root-ca.crt\""
Jul 31 15:18:20.633884 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.633760    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7w47\"/\"openshift-service-ca.crt\""
Jul 31 15:18:20.634750 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.634731    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l7w47\"/\"default-dockercfg-8djw8\""
Jul 31 15:18:20.640545 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.640526    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7w47/must-gather-t77fq"]
Jul 31 15:18:20.710253 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.710214    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e9a87f9-8918-4dc2-861e-c8977ae5febc-must-gather-output\") pod \"must-gather-t77fq\" (UID: \"8e9a87f9-8918-4dc2-861e-c8977ae5febc\") " pod="openshift-must-gather-l7w47/must-gather-t77fq"
Jul 31 15:18:20.710433 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.710340    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r76qv\" (UniqueName: \"kubernetes.io/projected/8e9a87f9-8918-4dc2-861e-c8977ae5febc-kube-api-access-r76qv\") pod \"must-gather-t77fq\" (UID: \"8e9a87f9-8918-4dc2-861e-c8977ae5febc\") " pod="openshift-must-gather-l7w47/must-gather-t77fq"
Jul 31 15:18:20.810958 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.810927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e9a87f9-8918-4dc2-861e-c8977ae5febc-must-gather-output\") pod \"must-gather-t77fq\" (UID: \"8e9a87f9-8918-4dc2-861e-c8977ae5febc\") " pod="openshift-must-gather-l7w47/must-gather-t77fq"
Jul 31 15:18:20.811165 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.810979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r76qv\" (UniqueName: \"kubernetes.io/projected/8e9a87f9-8918-4dc2-861e-c8977ae5febc-kube-api-access-r76qv\") pod \"must-gather-t77fq\" (UID: \"8e9a87f9-8918-4dc2-861e-c8977ae5febc\") " pod="openshift-must-gather-l7w47/must-gather-t77fq"
Jul 31 15:18:20.811308 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.811288    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8e9a87f9-8918-4dc2-861e-c8977ae5febc-must-gather-output\") pod \"must-gather-t77fq\" (UID: \"8e9a87f9-8918-4dc2-861e-c8977ae5febc\") " pod="openshift-must-gather-l7w47/must-gather-t77fq"
Jul 31 15:18:20.819110 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.819083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r76qv\" (UniqueName: \"kubernetes.io/projected/8e9a87f9-8918-4dc2-861e-c8977ae5febc-kube-api-access-r76qv\") pod \"must-gather-t77fq\" (UID: \"8e9a87f9-8918-4dc2-861e-c8977ae5febc\") " pod="openshift-must-gather-l7w47/must-gather-t77fq"
Jul 31 15:18:20.941311 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:20.941224    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7w47/must-gather-t77fq"
Jul 31 15:18:21.079603 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:21.079567    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7w47/must-gather-t77fq"]
Jul 31 15:18:21.080755 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:18:21.080730    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e9a87f9_8918_4dc2_861e_c8977ae5febc.slice/crio-c08093dee718f32eaa8d787d9c8bca46d744d580389d43921e1d2c9df2d6a177 WatchSource:0}: Error finding container c08093dee718f32eaa8d787d9c8bca46d744d580389d43921e1d2c9df2d6a177: Status 404 returned error can't find the container with id c08093dee718f32eaa8d787d9c8bca46d744d580389d43921e1d2c9df2d6a177
Jul 31 15:18:21.226906 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:21.226819    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/must-gather-t77fq" event={"ID":"8e9a87f9-8918-4dc2-861e-c8977ae5febc","Type":"ContainerStarted","Data":"c08093dee718f32eaa8d787d9c8bca46d744d580389d43921e1d2c9df2d6a177"}
Jul 31 15:18:22.237882 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:22.237848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/must-gather-t77fq" event={"ID":"8e9a87f9-8918-4dc2-861e-c8977ae5febc","Type":"ContainerStarted","Data":"0d27eac98f0f431e885a905958753d19d7d229fcc96a8dcd1c85931708f82430"}
Jul 31 15:18:23.248620 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:23.248584    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/must-gather-t77fq" event={"ID":"8e9a87f9-8918-4dc2-861e-c8977ae5febc","Type":"ContainerStarted","Data":"cee0739d6454e6ebbb36b54939e8b8452e21aa1bea82eba489012ed879dce361"}
Jul 31 15:18:23.265751 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:23.265686    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7w47/must-gather-t77fq" podStartSLOduration=2.2433185780000002 podStartE2EDuration="3.265664598s" totalImagesPullingTime="1.02234602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:23.263583398 +0000 UTC m=+3942.136146653" watchObservedRunningTime="2026-07-31 15:18:23.265664598 +0000 UTC m=+3942.138227853"
Jul 31 15:18:23.715838 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:23.715782    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sfw5j_1a778adb-72c6-427b-97e3-9da9685b0467/global-pull-secret-syncer/0.log"
Jul 31 15:18:23.881166 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:23.881127    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zlw8q_3d76362e-39cc-4a7a-88df-8902bf386dc1/konnectivity-agent/0.log"
Jul 31 15:18:23.954199 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:23.954165    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-143.ec2.internal_0fcb5a47836661934246ad400500ba6e/haproxy/0.log"
Jul 31 15:18:28.802138 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:28.802098    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-cl6tm_0f52ba2d-1e9b-44c3-816e-b257cdbca654/manager/0.log"
Jul 31 15:18:28.833308 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:28.833253    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-vvg9h_a1852a52-681d-48a7-9e03-564888a92c53/manager/0.log"
Jul 31 15:18:28.971044 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:28.969687    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-7n8ms_b1f1fcfc-a32b-4f95-837a-e4049c8400c4/manager/0.log"
Jul 31 15:18:29.160104 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:29.159589    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-bj8kz_47dc36c0-8bcf-46cc-935d-ea4749bcc9d1/manager/0.log"
Jul 31 15:18:30.733527 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:30.733486    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-87wx8_c5a24c7f-fb9e-4ead-b770-68d870cda0ed/cluster-monitoring-operator/0.log"
Jul 31 15:18:30.977379 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:30.977352    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfhj9_99cb1324-7f9a-45a4-9c2b-be4fc936ad44/node-exporter/0.log"
Jul 31 15:18:30.998627 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:30.998550    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfhj9_99cb1324-7f9a-45a4-9c2b-be4fc936ad44/kube-rbac-proxy/0.log"
Jul 31 15:18:31.023518 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:31.023489    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kfhj9_99cb1324-7f9a-45a4-9c2b-be4fc936ad44/init-textfile/0.log"
Jul 31 15:18:31.581177 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:31.581144    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57889dc56-7n8pq_624e76da-697c-4deb-8d42-12eecc3ff4a7/thanos-query/0.log"
Jul 31 15:18:31.603774 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:31.603739    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57889dc56-7n8pq_624e76da-697c-4deb-8d42-12eecc3ff4a7/kube-rbac-proxy-web/0.log"
Jul 31 15:18:31.629251 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:31.629218    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57889dc56-7n8pq_624e76da-697c-4deb-8d42-12eecc3ff4a7/kube-rbac-proxy/0.log"
Jul 31 15:18:31.650748 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:31.650713    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57889dc56-7n8pq_624e76da-697c-4deb-8d42-12eecc3ff4a7/prom-label-proxy/0.log"
Jul 31 15:18:31.672809 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:31.672777    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57889dc56-7n8pq_624e76da-697c-4deb-8d42-12eecc3ff4a7/kube-rbac-proxy-rules/0.log"
Jul 31 15:18:31.696187 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:31.696149    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57889dc56-7n8pq_624e76da-697c-4deb-8d42-12eecc3ff4a7/kube-rbac-proxy-metrics/0.log"
Jul 31 15:18:33.636840 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:33.636749    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/2.log"
Jul 31 15:18:33.647442 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:33.647410    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-ll4lr_5d416104-70a0-4285-bb1c-5ff965a4f9f4/console-operator/3.log"
Jul 31 15:18:34.250688 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.250651    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"]
Jul 31 15:18:34.259630 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.259591    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.259989 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.259963    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-c6s6l_8af6e796-b111-4aa2-91c9-77a2c7e7b6c0/download-server/0.log"
Jul 31 15:18:34.264444 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.264408    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"]
Jul 31 15:18:34.272356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.271343    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26r9\" (UniqueName: \"kubernetes.io/projected/42897203-1f53-4674-9a5d-88403aa8e5c1-kube-api-access-r26r9\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.272356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.271422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-proc\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.272356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.271448    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-podres\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.272356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.271536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-lib-modules\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.272356 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.271571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-sys\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.372184    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-proc\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.372238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-podres\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.372298    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-lib-modules\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.372327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-sys\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.372395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26r9\" (UniqueName: \"kubernetes.io/projected/42897203-1f53-4674-9a5d-88403aa8e5c1-kube-api-access-r26r9\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.372850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-proc\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.372980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-podres\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.373089    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-lib-modules\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.373679 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.373149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42897203-1f53-4674-9a5d-88403aa8e5c1-sys\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.384256 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.384224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26r9\" (UniqueName: \"kubernetes.io/projected/42897203-1f53-4674-9a5d-88403aa8e5c1-kube-api-access-r26r9\") pod \"perf-node-gather-daemonset-mbrz7\" (UID: \"42897203-1f53-4674-9a5d-88403aa8e5c1\") " pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.573997 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.573961    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:34.770148 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:34.770116    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"]
Jul 31 15:18:34.773126 ip-10-0-139-143 kubenswrapper[2558]: W0731 15:18:34.773089    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod42897203_1f53_4674_9a5d_88403aa8e5c1.slice/crio-e1ea50b35811ec6a8ad61f525d4495deee34b1471fdd8a7db30eb112341417a8 WatchSource:0}: Error finding container e1ea50b35811ec6a8ad61f525d4495deee34b1471fdd8a7db30eb112341417a8: Status 404 returned error can't find the container with id e1ea50b35811ec6a8ad61f525d4495deee34b1471fdd8a7db30eb112341417a8
Jul 31 15:18:35.010077 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:35.010048    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-nft87_6b83c90c-18c0-4778-8db6-7f8444419455/volume-data-source-validator/0.log"
Jul 31 15:18:35.364221 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:35.364177    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7" event={"ID":"42897203-1f53-4674-9a5d-88403aa8e5c1","Type":"ContainerStarted","Data":"085e007e5e44be4b9bf9557ddda0749d08517700409da1cac41ee904f37bfce2"}
Jul 31 15:18:35.364402 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:35.364261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7" event={"ID":"42897203-1f53-4674-9a5d-88403aa8e5c1","Type":"ContainerStarted","Data":"e1ea50b35811ec6a8ad61f525d4495deee34b1471fdd8a7db30eb112341417a8"}
Jul 31 15:18:35.365275 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:35.365246    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:35.383830 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:35.383769    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7" podStartSLOduration=1.383748824 podStartE2EDuration="1.383748824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:18:35.382293836 +0000 UTC m=+3954.254857090" watchObservedRunningTime="2026-07-31 15:18:35.383748824 +0000 UTC m=+3954.256312076"
Jul 31 15:18:36.189489 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:36.189452    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fmdw8_efbfc503-7eb7-435f-94a8-49b4fc9ecdd4/dns/0.log"
Jul 31 15:18:36.207530 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:36.207499    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fmdw8_efbfc503-7eb7-435f-94a8-49b4fc9ecdd4/kube-rbac-proxy/0.log"
Jul 31 15:18:36.313658 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:36.313633    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rtc4t_57c7c983-8b4d-4494-be24-c63071b5ad0a/dns-node-resolver/0.log"
Jul 31 15:18:36.882796 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:36.882757    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bcc48f99d-w2dpl_6b2cc95b-de81-47cf-a8d2-673a511e70fd/registry/0.log"
Jul 31 15:18:36.906831 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:36.906804    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gkw4n_66f05e4e-0197-4037-923c-c626b7982ab2/node-ca/0.log"
Jul 31 15:18:37.903558 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:37.903519    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c_4fa698e3-34cf-43d6-967b-492af65a098c/istio-proxy/0.log"
Jul 31 15:18:38.113681 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:38.113647    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8b44b796-zwgwr_80f5291b-3486-44ad-8db4-d9ce9301f825/kube-auth-proxy/0.log"
Jul 31 15:18:38.140773 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:38.140742    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-x79gg_7c5561a1-775e-4ccb-a096-19bbd398bde0/istio-proxy/0.log"
Jul 31 15:18:38.155126 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:38.155047    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-x79gg_7c5561a1-775e-4ccb-a096-19bbd398bde0/istio-proxy/1.log"
Jul 31 15:18:39.114640 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:39.114609    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m98v2_96a9dc2b-3f07-4be6-85a0-722f0b87c5ac/serve-healthcheck-canary/0.log"
Jul 31 15:18:39.580833 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:39.579374    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-lb47s_c6e38f19-a6c6-4eb8-909b-df5296e6655e/insights-operator/0.log"
Jul 31 15:18:39.580833 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:39.580350    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-lb47s_c6e38f19-a6c6-4eb8-909b-df5296e6655e/insights-operator/1.log"
Jul 31 15:18:39.599186 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:39.599159    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d8c7b_e366b3eb-b4bf-48a4-a97d-070b2b2c1977/kube-rbac-proxy/0.log"
Jul 31 15:18:39.617950 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:39.617920    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d8c7b_e366b3eb-b4bf-48a4-a97d-070b2b2c1977/exporter/0.log"
Jul 31 15:18:39.638244 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:39.638218    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d8c7b_e366b3eb-b4bf-48a4-a97d-070b2b2c1977/extractor/0.log"
Jul 31 15:18:41.942964 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:41.942922    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xzlhj_54829018-37a2-4699-af96-aea18ce0a6d3/manager/0.log"
Jul 31 15:18:42.368229 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:42.368199    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l65hb_ee99d4c6-3d42-42cd-a5d2-e6a94b651804/manager/0.log"
Jul 31 15:18:42.390613 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:42.390586    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l7w47/perf-node-gather-daemonset-mbrz7"
Jul 31 15:18:42.541804 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:42.541773    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-ptcph_c936181c-91ed-4e97-abc0-d63c2912a32a/server/0.log"
Jul 31 15:18:42.675097 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:42.674905    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-g95vk_b42a1bc2-e21f-44db-ab62-fb98ac94c8d1/manager/0.log"
Jul 31 15:18:42.816828 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:42.816793    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-k4dzm_67a9840e-4dcf-4ed4-8e17-9026c5b17cbe/manager/0.log"
Jul 31 15:18:42.838996 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:42.838963    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-698c94c45b-k4dzm_67a9840e-4dcf-4ed4-8e17-9026c5b17cbe/copy-manifests/0.log"
Jul 31 15:18:44.408927 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:44.408896    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-b58z8_ec566262-22c4-4de3-826e-fcc0c7777ef0/openshift-lws-operator/0.log"
Jul 31 15:18:44.802537 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:44.802500    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c_4fa698e3-34cf-43d6-967b-492af65a098c/istio-proxy/0.log"
Jul 31 15:18:44.985044 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:44.982132    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8b44b796-zwgwr_80f5291b-3486-44ad-8db4-d9ce9301f825/kube-auth-proxy/0.log"
Jul 31 15:18:45.013002 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:45.012971    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-x79gg_7c5561a1-775e-4ccb-a096-19bbd398bde0/istio-proxy/0.log"
Jul 31 15:18:45.025494 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:45.025327    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-x79gg_7c5561a1-775e-4ccb-a096-19bbd398bde0/istio-proxy/1.log"
Jul 31 15:18:46.569911 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:46.569854    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cvkf8c_4fa698e3-34cf-43d6-967b-492af65a098c/istio-proxy/0.log"
Jul 31 15:18:46.765153 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:46.765119    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8b44b796-zwgwr_80f5291b-3486-44ad-8db4-d9ce9301f825/kube-auth-proxy/0.log"
Jul 31 15:18:46.788423 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:46.788388    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-x79gg_7c5561a1-775e-4ccb-a096-19bbd398bde0/istio-proxy/0.log"
Jul 31 15:18:46.802303 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:46.802275    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-x79gg_7c5561a1-775e-4ccb-a096-19bbd398bde0/istio-proxy/1.log"
Jul 31 15:18:49.440803 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:49.440768    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-z9jlm_ea8f38c1-1fef-4593-bf10-f1f954c384ce/migrator/0.log"
Jul 31 15:18:49.460521 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:49.460495    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-z9jlm_ea8f38c1-1fef-4593-bf10-f1f954c384ce/graceful-termination/0.log"
Jul 31 15:18:49.830828 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:49.830797    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-pd68z_6aeed834-fbee-468e-ab99-ac7121d9fb08/kube-storage-version-migrator-operator/1.log"
Jul 31 15:18:49.832567 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:49.832540    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-pd68z_6aeed834-fbee-468e-ab99-ac7121d9fb08/kube-storage-version-migrator-operator/0.log"
Jul 31 15:18:51.112094 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.112069    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wp2xh_37ea6151-ad56-41c1-8ed5-ede42ab94632/kube-multus-additional-cni-plugins/0.log"
Jul 31 15:18:51.130517 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.130492    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wp2xh_37ea6151-ad56-41c1-8ed5-ede42ab94632/egress-router-binary-copy/0.log"
Jul 31 15:18:51.150317 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.150293    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wp2xh_37ea6151-ad56-41c1-8ed5-ede42ab94632/cni-plugins/0.log"
Jul 31 15:18:51.167827 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.167798    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wp2xh_37ea6151-ad56-41c1-8ed5-ede42ab94632/bond-cni-plugin/0.log"
Jul 31 15:18:51.185176 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.185142    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wp2xh_37ea6151-ad56-41c1-8ed5-ede42ab94632/routeoverride-cni/0.log"
Jul 31 15:18:51.202838 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.202811    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wp2xh_37ea6151-ad56-41c1-8ed5-ede42ab94632/whereabouts-cni-bincopy/0.log"
Jul 31 15:18:51.221748 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.221723    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wp2xh_37ea6151-ad56-41c1-8ed5-ede42ab94632/whereabouts-cni/0.log"
Jul 31 15:18:51.283522 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.283490    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wccvh_83810d6e-f7ae-46d0-a38c-4bab260b035d/kube-multus/0.log"
Jul 31 15:18:51.341175 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.341147    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jbbns_13d90876-b172-48d8-95e3-420e703c71cf/network-metrics-daemon/0.log"
Jul 31 15:18:51.354874 ip-10-0-139-143 kubenswrapper[2558]: I0731 15:18:51.354838    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jbbns_13d90876-b172-48d8-95e3-420e703c71cf/kube-rbac-proxy/0.log"