Jul 22 02:28:28.640622 ip-10-0-133-30 systemd[1]: Starting Kubernetes Kubelet...
Jul 22 02:28:29.025830 ip-10-0-133-30 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 02:28:29.025830 ip-10-0-133-30 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 22 02:28:29.025830 ip-10-0-133-30 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 02:28:29.025830 ip-10-0-133-30 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 22 02:28:29.025830 ip-10-0-133-30 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 02:28:29.027742 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.027591    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 22 02:28:29.036776 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036746    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:28:29.036776 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036770    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:28:29.036776 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036774    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:28:29.036776 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036778    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:28:29.036776 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036781    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:28:29.036776 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036784    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:28:29.036776 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036787    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036790    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036792    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036795    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036798    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036800    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036803    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036806    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036808    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036810    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036813    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036818    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036821    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036824    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036827    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036830    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036833    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036836    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036838    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:28:29.037037 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036841    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036846    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036849    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036852    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036854    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036857    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036859    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036861    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036864    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036866    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036869    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036871    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036874    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036876    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036879    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036882    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036884    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036889    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036893    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:28:29.037540 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036896    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036899    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036902    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036904    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036907    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036910    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036913    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036920    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036923    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036926    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036928    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036931    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036933    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036936    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036939    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036941    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036944    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036946    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036949    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036951    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:28:29.038020 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036953    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036956    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036958    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036961    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036963    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036966    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036968    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036973    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036979    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036982    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036984    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036987    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036990    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036992    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036995    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.036998    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037000    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037003    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037005    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:28:29.038507 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037008    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037010    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037013    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037015    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037017    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037020    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037022    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037025    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037027    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037030    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037032    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037034    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037037    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037039    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037042    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037045    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037801    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037808    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037812    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037815    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:28:29.038966 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037818    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037822    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037825    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037827    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037830    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037833    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037835    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037838    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037840    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037843    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037845    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037848    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037851    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037857    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037859    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037862    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037865    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037868    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037871    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037873    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:28:29.039511 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037876    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037878    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037882    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037885    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037887    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037890    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037893    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037895    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037898    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037901    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037903    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037906    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037908    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037911    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037913    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037916    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037918    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037920    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037924    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037926    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:28:29.040034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037929    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037933    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037937    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037940    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037943    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037945    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037949    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037951    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037954    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037956    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037959    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037962    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037965    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037967    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037970    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037973    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037975    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037978    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037980    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:28:29.040533 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037984    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037987    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037990    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037992    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037995    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.037998    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038002    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038005    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038008    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038011    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038014    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038016    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038019    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038022    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038025    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038027    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038030    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038033    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038036    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:28:29.040981 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038039    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038041    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038044    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038046    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038049    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038051    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038054    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038057    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038060    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038063    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038066    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038069    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038071    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038074    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038076    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038079    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038081    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038487    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038492    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038495    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:28:29.041471 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038498    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038501    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038503    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038506    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038509    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038512    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038514    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038517    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038520    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038523    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038525    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038528    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038530    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038534    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038537    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038539    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038542    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038544    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038549    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:28:29.041972 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038552    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038557    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038560    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038563    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038567    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038570    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038572    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038575    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038578    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038581    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038584    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038587    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038590    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038592    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038595    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038597    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038600    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038602    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038605    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:28:29.042453 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038607    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038609    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038613    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038616    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038619    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038621    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038624    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038626    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038629    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038631    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038634    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038637    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038641    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038644    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038647    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038649    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038652    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038655    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038657    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038660    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:28:29.042905 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038663    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038665    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038667    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038670    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038673    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038675    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038677    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038680    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038683    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038685    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038687    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038690    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038693    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038695    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038697    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038700    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038702    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038705    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038708    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038710    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:28:29.043401 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038713    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038715    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038718    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038721    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038725    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038727    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038730    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038732    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038734    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038737    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038740    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038745    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038748    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038752    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038754    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038757    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038760    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.038763    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038838    2583 flags.go:64] FLAG: --address="0.0.0.0"
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038846    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038852    2583 flags.go:64] FLAG: --anonymous-auth="true"
Jul 22 02:28:29.043901 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038856    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038860    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038863    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038867    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038872    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038875    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038878    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038882    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038886    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038890    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038893    2583 flags.go:64] FLAG: --cgroup-root=""
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038896    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038899    2583 flags.go:64] FLAG: --client-ca-file=""
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038903    2583 flags.go:64] FLAG: --cloud-provider="external"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038906    2583 flags.go:64] FLAG: --cluster-dns="[]"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038913    2583 flags.go:64] FLAG: --cluster-domain=""
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038916    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038919    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038922    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038925    2583 flags.go:64] FLAG: --container-log-max-files="5"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038929    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038932    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038935    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 22 02:28:29.044485 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038938    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038941    2583 flags.go:64] FLAG: --contention-profiling="false"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038943    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038946    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038949    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038952    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038956    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038959    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038962    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038965    2583 flags.go:64] FLAG: --enable-load-reader="false"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038968    2583 flags.go:64] FLAG: --enable-server="true"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038971    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038975    2583 flags.go:64] FLAG: --event-burst="100"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038978    2583 flags.go:64] FLAG: --event-qps="50"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038981    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038984    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038987    2583 flags.go:64] FLAG: --eviction-hard=""
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038991    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038994    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.038997    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039000    2583 flags.go:64] FLAG: --eviction-soft=""
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039003    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039006    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039009    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039013    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 22 02:28:29.045031 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039016    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039019    2583 flags.go:64] FLAG: --fail-swap-on="true"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039022    2583 flags.go:64] FLAG: --feature-gates=""
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039025    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039028    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039031    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039034    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039037    2583 flags.go:64] FLAG: --healthz-port="10248"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039040    2583 flags.go:64] FLAG: --help="false"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039043    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039046    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039049    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039051    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039055    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039058    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039060    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039063    2583 flags.go:64] FLAG: --image-service-endpoint=""
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039066    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039069    2583 flags.go:64] FLAG: --kube-api-burst="100"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039071    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039074    2583 flags.go:64] FLAG: --kube-api-qps="50"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039077    2583 flags.go:64] FLAG: --kube-reserved=""
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039080    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039083    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 22 02:28:29.045720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039086    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039089    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039092    2583 flags.go:64] FLAG: --lock-file=""
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039094    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039097    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039100    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039105    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039110    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039113    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039116    2583 flags.go:64] FLAG: --logging-format="text"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039119    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039122    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039124    2583 flags.go:64] FLAG: --manifest-url=""
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039127    2583 flags.go:64] FLAG: --manifest-url-header=""
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039131    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039134    2583 flags.go:64] FLAG: --max-open-files="1000000"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039138    2583 flags.go:64] FLAG: --max-pods="110"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039141    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039144    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039146    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039150    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039152    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039155    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039158    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039165    2583 flags.go:64] FLAG: --node-status-max-images="50"
Jul 22 02:28:29.046289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039168    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039171    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039174    2583 flags.go:64] FLAG: --pod-cidr=""
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039176    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039182    2583 flags.go:64] FLAG: --pod-manifest-path=""
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039184    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039187    2583 flags.go:64] FLAG: --pods-per-core="0"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039190    2583 flags.go:64] FLAG: --port="10250"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039193    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039196    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0040f4672372181bf"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039199    2583 flags.go:64] FLAG: --qos-reserved=""
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039202    2583 flags.go:64] FLAG: --read-only-port="10255"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039205    2583 flags.go:64] FLAG: --register-node="true"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039207    2583 flags.go:64] FLAG: --register-with-taints=""
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039211    2583 flags.go:64] FLAG: --registry-burst="10"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039227    2583 flags.go:64] FLAG: --registry-qps="5"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039230    2583 flags.go:64] FLAG: --reserved-cpus=""
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039233    2583 flags.go:64] FLAG: --reserved-memory=""
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039237    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039240    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039243    2583 flags.go:64] FLAG: --rotate-certificates="false"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039246    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039249    2583 flags.go:64] FLAG: --runonce="false"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039251    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039254    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 22 02:28:29.046920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039257    2583 flags.go:64] FLAG: --seccomp-default="false"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039260    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039263    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039266    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039268    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039271    2583 flags.go:64] FLAG: --storage-driver-password="root"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039274    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039277    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039280    2583 flags.go:64] FLAG: --storage-driver-user="root"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039282    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039285    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039288    2583 flags.go:64] FLAG: --system-cgroups=""
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039291    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039296    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039298    2583 flags.go:64] FLAG: --tls-cert-file=""
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039301    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039306    2583 flags.go:64] FLAG: --tls-min-version=""
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039308    2583 flags.go:64] FLAG: --tls-private-key-file=""
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039311    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039332    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039336    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039341    2583 flags.go:64] FLAG: --v="2"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039348    2583 flags.go:64] FLAG: --version="false"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039352    2583 flags.go:64] FLAG: --vmodule=""
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039356    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 22 02:28:29.047551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039360    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039495    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039500    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039503    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039506    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039508    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039511    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039514    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039517    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039520    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039522    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039525    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039528    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039531    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039534    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039536    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039539    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039542    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039544    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039547    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:28:29.048161 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039549    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039553    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039556    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039558    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039561    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039563    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039566    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039568    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039572    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039577    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039580    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039583    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039586    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039589    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039591    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039594    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039596    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039599    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039601    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:28:29.048673 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039604    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039606    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039608    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039611    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039613    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039616    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039618    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039621    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039623    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039626    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039628    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039631    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039633    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039636    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039639    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039642    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039645    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039647    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039650    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039652    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:28:29.049189 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039655    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039657    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039660    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039662    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039665    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039668    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039670    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039673    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039675    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039678    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039680    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039683    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039685    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039687    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039691    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039694    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039698    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039700    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039703    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:28:29.049703 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039705    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039708    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039710    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039713    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039715    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039718    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039720    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039723    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039726    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039729    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039731    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039733    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039736    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039738    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039741    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039743    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039745    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039748    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039751    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:28:29.050165 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039754    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039757    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.039760    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.039766    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.046792    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.046809    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.047683    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.047690    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047771    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047777    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047781    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047786    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047789    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047792    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:28:29.050693 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047795    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047798    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047801    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047803    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047806    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047809    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047812    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047814    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047817    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047820    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047823    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047825    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047827    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047830    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047833    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047836    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047838    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047840    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047844    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:28:29.051034 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047847    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047849    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047852    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047855    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047857    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047860    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047863    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047865    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047868    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047870    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047873    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047875    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047878    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047880    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047883    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047886    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047889    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047893    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047896    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047899    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:28:29.051514 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047902    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047905    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047907    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047910    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047912    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047915    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047917    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047919    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047922    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047924    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047927    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047929    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047932    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047934    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047937    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047940    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047943    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047946    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047948    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047951    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:28:29.052007 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047953    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047956    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047958    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047960    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047963    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047965    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047968    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047971    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047973    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047976    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047978    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047981    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047983    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047986    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047988    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047990    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047993    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047995    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.047998    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048000    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:28:29.052505 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048002    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048005    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048007    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048010    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048013    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048021    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048024    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048027    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048030    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048034    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048037    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048040    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048043    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048045    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.048050    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 02:28:29.053013 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048187    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048192    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048195    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048197    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048200    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048203    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048205    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048208    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048211    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048214    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048217    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048219    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048222    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048225    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048227    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048230    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048232    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048235    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048237    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048240    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 02:28:29.053413 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048242    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048245    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048247    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048255    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048258    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048261    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048264    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048266    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048269    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048271    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048273    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048276    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048278    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048280    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048283    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048285    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048287    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048290    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048292    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048295    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 02:28:29.053894 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048297    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048299    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048301    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048304    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048306    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048309    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048311    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048326    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048329    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048331    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048334    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048336    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048339    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048341    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048344    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048347    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048355    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048358    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048360    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048363    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 02:28:29.054517 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048366    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048368    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048371    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048373    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048376    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048379    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048381    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048385    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048388    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048391    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048394    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048397    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048399    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048402    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048405    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048407    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048409    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048412    2583 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048415    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 02:28:29.055026 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048417    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048420    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048422    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048425    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048427    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048429    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048432    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048434    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048436    2583 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048439    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048447    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048450    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048452    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048456    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048459    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048462    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048465    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048468    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048471    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 02:28:29.055530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:29.048473    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 02:28:29.055998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.048478    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 02:28:29.055998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.048599    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 22 02:28:29.055998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.050632    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 22 02:28:29.055998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.052069    2583 server.go:1019] "Starting client certificate rotation"
Jul 22 02:28:29.055998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.052186    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 02:28:29.055998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.052222    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 02:28:29.055998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.054657    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 02:28:29.057607 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.057573    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 02:28:29.067703 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.067678    2583 log.go:25] "Validated CRI v1 runtime API"
Jul 22 02:28:29.070679 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.070662    2583 log.go:25] "Validated CRI v1 image API"
Jul 22 02:28:29.072231 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.072210    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 22 02:28:29.074824 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.074805    2583 fs.go:135] Filesystem UUIDs: map[2ca9eda7-e208-457c-8cc4-6f6998987ce9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8e220be9-5dd8-44d5-9667-8559350840fd:/dev/nvme0n1p4]
Jul 22 02:28:29.074883 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.074823    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 22 02:28:29.079993 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.079875    2583 manager.go:216] Machine: {Timestamp:2026-07-22 02:28:29.078476211 +0000 UTC m=+0.324921723 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100956 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bfc0d154e20faf28bd8aaca80c12a SystemUUID:ec2bfc0d-154e-20fa-f28b-d8aaca80c12a BootID:da8434c9-1795-4fdf-a7ac-cf1b1a235d01 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:26:0d:e0:51:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:26:0d:e0:51:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:bc:8a:ae:ca:d3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 22 02:28:29.079993 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.079985    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 22 02:28:29.080111 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.080066    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 22 02:28:29.084032 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.083998    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 22 02:28:29.084175 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.084033    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 22 02:28:29.084691 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.084681    2583 topology_manager.go:138] "Creating topology manager with none policy"
Jul 22 02:28:29.084723 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.084693    2583 container_manager_linux.go:309] "Creating device plugin manager"
Jul 22 02:28:29.084723 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.084700    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 02:28:29.085391 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.085380    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 02:28:29.086555 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.086545    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 22 02:28:29.089606 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.089592    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 02:28:29.089725 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.089715    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 22 02:28:29.089778 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.089745    2583 kubelet.go:486] "Attempting to sync node with API server"
Jul 22 02:28:29.089778 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.089756    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 22 02:28:29.089865 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.089787    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 22 02:28:29.089865 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.089799    2583 kubelet.go:398] "Adding apiserver pod source"
Jul 22 02:28:29.089865 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.089808    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 22 02:28:29.090210 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.090198    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 02:28:29.090259 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.090217    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 02:28:29.091088 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.091071    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 02:28:29.093622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.093600    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 22 02:28:29.097247 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.097226    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 22 02:28:29.097360 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.097270    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 22 02:28:29.098961 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.098940    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.098966    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.098976    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.098985    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.098995    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099002    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099010    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099015    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099022    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099027    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099036    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 22 02:28:29.099043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099041    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 22 02:28:29.099679 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099666    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 22 02:28:29.099679 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.099679    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 22 02:28:29.101469 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.101439    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 22 02:28:29.101469 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.101438    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 22 02:28:29.101694 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.101676    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 22 02:28:29.103027 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.103011    2583 server.go:1268] "Started kubelet"
Jul 22 02:28:29.103112 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.103078    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 22 02:28:29.103205 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.103147    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 22 02:28:29.103302 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.103228    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 22 02:28:29.103937 ip-10-0-133-30 systemd[1]: Started Kubernetes Kubelet.
Jul 22 02:28:29.104821 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.104765    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 22 02:28:29.104955 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.104948    2583 server.go:310] "Adding debug handlers to kubelet server"
Jul 22 02:28:29.109251 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.109233    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 22 02:28:29.111492 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.111472    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 22 02:28:29.111582 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.111474    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 22 02:28:29.111730 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.111705    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.112203 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112184    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 22 02:28:29.112300 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112201    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 22 02:28:29.112300 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112204    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 22 02:28:29.112430 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112345    2583 reconstruct.go:97] "Volume reconstruction finished"
Jul 22 02:28:29.112430 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112359    2583 reconciler.go:29] "Reconciler: start to sync state"
Jul 22 02:28:29.112430 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112402    2583 factory.go:55] Registering systemd factory
Jul 22 02:28:29.112430 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112421    2583 factory.go:223] Registration of the systemd container factory successfully
Jul 22 02:28:29.112688 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112674    2583 factory.go:153] Registering CRI-O factory
Jul 22 02:28:29.112753 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112694    2583 factory.go:223] Registration of the crio container factory successfully
Jul 22 02:28:29.112753 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112743    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 22 02:28:29.112825 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112764    2583 factory.go:103] Registering Raw factory
Jul 22 02:28:29.112825 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.112778    2583 manager.go:1196] Started watching for new ooms in manager
Jul 22 02:28:29.113980 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.113958    2583 manager.go:349] Starting recovery of all containers
Jul 22 02:28:29.116821 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.116664    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 22 02:28:29.116821 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.116667    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 22 02:28:29.118139 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.116756    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-30.ec2.internal.18c47ba6e63b23e2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-30.ec2.internal,UID:ip-10-0-133-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-30.ec2.internal,},FirstTimestamp:2026-07-22 02:28:29.102982114 +0000 UTC m=+0.349427628,LastTimestamp:2026-07-22 02:28:29.102982114 +0000 UTC m=+0.349427628,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-30.ec2.internal,}"
Jul 22 02:28:29.124733 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.124577    2583 manager.go:354] Recovery completed
Jul 22 02:28:29.126883 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.126865    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gckc2"
Jul 22 02:28:29.130010 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.129998    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:28:29.133078 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.133060    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gckc2"
Jul 22 02:28:29.133913 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.133896    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:28:29.133985 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.133926    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:28:29.133985 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.133937    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:28:29.134475 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.134459    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 22 02:28:29.134475 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.134472    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 22 02:28:29.134586 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.134488    2583 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 02:28:29.134586 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.134465    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:28:29.135270 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.135253    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:28:29.135361 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.135284    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:28:29.135361 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.135307    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:28:29.135729 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.135717    2583 policy_none.go:49] "None policy: Start"
Jul 22 02:28:29.135778 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.135734    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 22 02:28:29.135778 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.135744    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 22 02:28:29.136244 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.136170    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-30.ec2.internal.18c47ba6e8131a7e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-30.ec2.internal,UID:ip-10-0-133-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-30.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-30.ec2.internal,},FirstTimestamp:2026-07-22 02:28:29.133912702 +0000 UTC m=+0.380358214,LastTimestamp:2026-07-22 02:28:29.133912702 +0000 UTC m=+0.380358214,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-30.ec2.internal,}"
Jul 22 02:28:29.136964 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.136952    2583 policy_none.go:47] "Start"
Jul 22 02:28:29.175980 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.175961    2583 manager.go:337] "Starting Device Plugin manager"
Jul 22 02:28:29.176098 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.176003    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 22 02:28:29.176098 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.176012    2583 server.go:87] "Starting device plugin registration server"
Jul 22 02:28:29.176340 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.176310    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 22 02:28:29.176387 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.176342    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 22 02:28:29.176451 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.176433    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 22 02:28:29.176529 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.176515    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 22 02:28:29.176529 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.176528    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 22 02:28:29.177023 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.177005    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 22 02:28:29.177086 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.177055    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.276742 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.276643    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:28:29.278038 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.277701    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:28:29.278038 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.277734    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:28:29.278038 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.277751    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:28:29.278038 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.277782    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.285953 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.285938    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.285999 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.285961    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-30.ec2.internal\": node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.295307 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.295280    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 22 02:28:29.295696 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.295675    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.296738 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.296723    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 22 02:28:29.296789 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.296760    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 22 02:28:29.296789 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.296784    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 22 02:28:29.296858 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.296822    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 22 02:28:29.298894 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.298875    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:28:29.396730 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.396684    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.397791 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.397753    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal"]
Jul 22 02:28:29.397846 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.397837    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:28:29.398872 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.398858    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:28:29.398930 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.398888    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:28:29.398930 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.398901    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:28:29.400229 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.400217    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:28:29.400378 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.400362    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.400416 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.400397    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:28:29.400930 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.400913    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:28:29.401008 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.400943    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:28:29.401008 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.400953    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:28:29.401008 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.400913    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:28:29.401095 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.401019    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:28:29.401095 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.401030    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:28:29.402218 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.402205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.402260 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.402231    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 02:28:29.402890 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.402874    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 02:28:29.402944 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.402911    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 02:28:29.402944 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.402925    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeHasSufficientPID"
Jul 22 02:28:29.426119 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.426091    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-30.ec2.internal\" not found" node="ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.430439 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.430410    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-30.ec2.internal\" not found" node="ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.497713 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.497685    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.513590 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.513542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2721bad79031ea0bab955dc2fb6d89c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal\" (UID: \"d2721bad79031ea0bab955dc2fb6d89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.513730 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.513598    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2721bad79031ea0bab955dc2fb6d89c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal\" (UID: \"d2721bad79031ea0bab955dc2fb6d89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.513730 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.513648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc224443957a17fe5223c8975763ab96-config\") pod \"kube-apiserver-proxy-ip-10-0-133-30.ec2.internal\" (UID: \"bc224443957a17fe5223c8975763ab96\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.598832 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.598743    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.614114 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.614090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2721bad79031ea0bab955dc2fb6d89c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal\" (UID: \"d2721bad79031ea0bab955dc2fb6d89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.614199 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.614120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2721bad79031ea0bab955dc2fb6d89c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal\" (UID: \"d2721bad79031ea0bab955dc2fb6d89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.614199 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.614142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc224443957a17fe5223c8975763ab96-config\") pod \"kube-apiserver-proxy-ip-10-0-133-30.ec2.internal\" (UID: \"bc224443957a17fe5223c8975763ab96\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.614293 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.614175    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc224443957a17fe5223c8975763ab96-config\") pod \"kube-apiserver-proxy-ip-10-0-133-30.ec2.internal\" (UID: \"bc224443957a17fe5223c8975763ab96\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.614293 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.614227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2721bad79031ea0bab955dc2fb6d89c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal\" (UID: \"d2721bad79031ea0bab955dc2fb6d89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.614293 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.614248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2721bad79031ea0bab955dc2fb6d89c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal\" (UID: \"d2721bad79031ea0bab955dc2fb6d89c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.699296 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.699264    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.729765 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.729736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.733929 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:29.733755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:29.800375 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.800309    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:29.900891 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:29.900820    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:31.567153 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:31.567108    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:31.617609 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:31.617582    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 22 02:28:31.618063 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:31.617746    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 22 02:28:31.668167 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:31.668113    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:31.703295 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:31.703249    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 02:23:30 +0000 UTC" deadline="2028-03-19 15:38:21.739858414 +0000 UTC"
Jul 22 02:28:31.703295 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:31.703286    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14557h9m50.036577301s"
Jul 22 02:28:31.768613 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:31.768586    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:31.869620 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:31.869592    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:31.883551 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:31.883514    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2721bad79031ea0bab955dc2fb6d89c.slice/crio-ffd431b805b49a2d0ea84e18ad5d60b5dcdf43949d628b6238b09fda645ca828 WatchSource:0}: Error finding container ffd431b805b49a2d0ea84e18ad5d60b5dcdf43949d628b6238b09fda645ca828: Status 404 returned error can't find the container with id ffd431b805b49a2d0ea84e18ad5d60b5dcdf43949d628b6238b09fda645ca828
Jul 22 02:28:31.883869 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:31.883850    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc224443957a17fe5223c8975763ab96.slice/crio-fe0cbc062b0099cf26037aa3414048db2f0099a7a08fce375d789cd2c05a5f74 WatchSource:0}: Error finding container fe0cbc062b0099cf26037aa3414048db2f0099a7a08fce375d789cd2c05a5f74: Status 404 returned error can't find the container with id fe0cbc062b0099cf26037aa3414048db2f0099a7a08fce375d789cd2c05a5f74
Jul 22 02:28:31.889829 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:31.889816    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:28:31.902255 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:31.902232    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:28:31.959872 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:31.959834    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:28:31.969726 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:31.969698    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:32.070301 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.070224    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:32.171068 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.171010    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-30.ec2.internal\" not found"
Jul 22 02:28:32.219234 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.219202    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:28:32.277935 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.277911    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:32.290965 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.290937    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 02:28:32.292136 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.292114    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal"
Jul 22 02:28:32.302096 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.302074    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 02:28:32.596727 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.596698    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 02:28:32.656218 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.656178    2583 apiserver.go:52] "Watching apiserver"
Jul 22 02:28:32.664513 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.664483    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 22 02:28:32.666601 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.666573    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-sb2l5","openshift-dns/node-resolver-t2hdl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal","openshift-multus/multus-7xjsv","openshift-multus/multus-additional-cni-plugins-zfxnf","openshift-network-diagnostics/network-check-target-4gnks","kube-system/konnectivity-agent-qpnz2","kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal","openshift-image-registry/node-ca-w8jgr","openshift-multus/network-metrics-daemon-5t4kv","openshift-network-operator/iptables-alerter-6hj7x","openshift-ovn-kubernetes/ovnkube-node-xxrcb","kube-system/global-pull-secret-syncer-6grr5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"]
Jul 22 02:28:32.668080 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.668053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:32.669918 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.669892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.670845 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.670827    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 22 02:28:32.670941 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.670856    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 22 02:28:32.671232 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.671212    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d88sn\""
Jul 22 02:28:32.671564 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.671546    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.671924 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.671907    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 22 02:28:32.672423 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.672407    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-47dqn\""
Jul 22 02:28:32.672804 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.672789    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 22 02:28:32.672966 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.672953    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 22 02:28:32.673298 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.673278    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.673958 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.673930    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 22 02:28:32.673958 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.673958    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-828xl\""
Jul 22 02:28:32.674238 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.674220    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 22 02:28:32.674367 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.674247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.674427 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.674384    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 22 02:28:32.675358 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.675340    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 22 02:28:32.675475 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.675389    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:32.675475 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.675448    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:32.675779 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.675763    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 22 02:28:32.676390 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.676369    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k6wz2\""
Jul 22 02:28:32.676868 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.676695    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 22 02:28:32.676934 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.676895    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 22 02:28:32.677161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.677121    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.677680 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.677658    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 22 02:28:32.677801 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.677781    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fbh22\""
Jul 22 02:28:32.677867 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.677784    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 22 02:28:32.678009 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.677789    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 22 02:28:32.679383 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.679361    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 22 02:28:32.679566 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.679543    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zpbfz\""
Jul 22 02:28:32.679812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.679791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.680074 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.680053    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 22 02:28:32.680515 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.680495    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 22 02:28:32.681487 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.681468    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.683141 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.683117    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 22 02:28:32.683373 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.683357    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 22 02:28:32.684020 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.684003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:32.684020 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.684090    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:32.684525 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.684510    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 22 02:28:32.684780 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.684763    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 22 02:28:32.684945 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.684932    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 22 02:28:32.685185 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.685169    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jw58b\""
Jul 22 02:28:32.685404 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.685389    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 22 02:28:32.685619 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.685605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hs8ng\""
Jul 22 02:28:32.685841 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.685827    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 22 02:28:32.685928 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.685913    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.685985 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.685973    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:32.686076 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.686063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.686076 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.686070    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2e694d6-f4a6-48bf-a27c-219fcd70e25c-agent-certs\") pod \"konnectivity-agent-qpnz2\" (UID: \"e2e694d6-f4a6-48bf-a27c-219fcd70e25c\") " pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-systemd\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-var-lib-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-var-lib-kubelet\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-tmp\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697443    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-registration-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697476    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-modprobe-d\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697534    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-cni-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-etc-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697592    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-sys\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697595    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03b319fd-30b2-46a9-9983-769ec8187f99-iptables-alerter-script\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-socket-dir-parent\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697740    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-systemd\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697809    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-ovn\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.698810 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-serviceca\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-slash\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-sys-fs\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-kubernetes\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysctl-conf\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-tuned\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-conf-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-multus-certs\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698158    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxfs\" (UniqueName: \"kubernetes.io/projected/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-kube-api-access-qgxfs\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcs4k\" (UniqueName: \"kubernetes.io/projected/03b319fd-30b2-46a9-9983-769ec8187f99-kube-api-access-fcs4k\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-os-release\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-daemon-config\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swpkl\" (UniqueName: \"kubernetes.io/projected/446f6070-adf6-4559-b8e1-d0e7fc6fff57-kube-api-access-swpkl\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.697727    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ksbpp\""
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-cni-bin\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698671    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.699812 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-etc-kubernetes\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698747    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-systemd-units\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-node-log\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-cni-netd\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-hostroot\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-device-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.698980    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b319fd-30b2-46a9-9983-769ec8187f99-host-slash\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-kubelet\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2e694d6-f4a6-48bf-a27c-219fcd70e25c-konnectivity-ca\") pod \"konnectivity-agent-qpnz2\" (UID: \"e2e694d6-f4a6-48bf-a27c-219fcd70e25c\") " pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699091    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-host\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78kmd\" (UniqueName: \"kubernetes.io/projected/eec10c29-003e-489a-af40-bdb5844bfc14-kube-api-access-78kmd\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-k8s-cni-cncf-io\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysctl-d\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.700618 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-run\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699386    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-cnibin\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/446f6070-adf6-4559-b8e1-d0e7fc6fff57-cni-binary-copy\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdj5h\" (UniqueName: \"kubernetes.io/projected/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-kube-api-access-cdj5h\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-ovnkube-config\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-env-overrides\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-socket-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-kubelet\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699750    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-ovnkube-script-lib\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysconfig\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-lib-modules\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-log-socket\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.699956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-system-cni-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.700010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-netns\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.700084    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b75189cc-3291-4d83-85d9-c5590098fea5-ovn-node-metrics-cert\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.700140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-cni-bin\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.700184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-cni-multus\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.701356 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.700244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-host\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.702212 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.700278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-run-netns\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.702212 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.700316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5mk\" (UniqueName: \"kubernetes.io/projected/b75189cc-3291-4d83-85d9-c5590098fea5-kube-api-access-kf5mk\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.702212 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.701115    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 02:28:32.717117 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.716805    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-shls6"
Jul 22 02:28:32.728748 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.728722    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-shls6"
Jul 22 02:28:32.778926 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.778899    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 22 02:28:32.801238 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-kubelet-config\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2e694d6-f4a6-48bf-a27c-219fcd70e25c-agent-certs\") pod \"konnectivity-agent-qpnz2\" (UID: \"e2e694d6-f4a6-48bf-a27c-219fcd70e25c\") " pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-systemd\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-var-lib-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801348    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-var-lib-kubelet\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801362    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-tmp\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801378    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-registration-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98d6\" (UniqueName: \"kubernetes.io/projected/f5aed715-01ac-4ee0-a911-aae835161a19-kube-api-access-t98d6\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.801420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-modprobe-d\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801433    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-cni-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-etc-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-dbus\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-sys\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03b319fd-30b2-46a9-9983-769ec8187f99-iptables-alerter-script\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-socket-dir-parent\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801551    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-os-release\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-cni-binary-copy\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-tmp-dir\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-systemd\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-ovn\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-serviceca\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-slash\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-sys-fs\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-kubernetes\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.801907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysctl-conf\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-tuned\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801779    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-conf-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-multus-certs\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxfs\" (UniqueName: \"kubernetes.io/projected/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-kube-api-access-qgxfs\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcs4k\" (UniqueName: \"kubernetes.io/projected/03b319fd-30b2-46a9-9983-769ec8187f99-kube-api-access-fcs4k\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801840    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-os-release\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-daemon-config\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swpkl\" (UniqueName: \"kubernetes.io/projected/446f6070-adf6-4559-b8e1-d0e7fc6fff57-kube-api-access-swpkl\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-cni-bin\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-etc-kubernetes\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-systemd-units\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-node-log\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.801995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-cni-netd\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-hostroot\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.804469 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtcnp\" (UniqueName: \"kubernetes.io/projected/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-kube-api-access-wtcnp\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-device-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b319fd-30b2-46a9-9983-769ec8187f99-host-slash\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802147    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-socket-dir-parent\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-kubelet\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-sys-fs\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-kubelet\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-systemd-units\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802201    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-modprobe-d\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2e694d6-f4a6-48bf-a27c-219fcd70e25c-konnectivity-ca\") pod \"konnectivity-agent-qpnz2\" (UID: \"e2e694d6-f4a6-48bf-a27c-219fcd70e25c\") " pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-cni-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-ovn\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-host\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802248    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-node-log\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-etc-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802296    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-host\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.805274 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-cni-netd\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kmd\" (UniqueName: \"kubernetes.io/projected/eec10c29-003e-489a-af40-bdb5844bfc14-kube-api-access-78kmd\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802340    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-sys\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-kubernetes\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802351    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-k8s-cni-cncf-io\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802391    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802397    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-device-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysctl-d\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-os-release\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-run\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-cnibin\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-slash\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-k8s-cni-cncf-io\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802392    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-multus-certs\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/03b319fd-30b2-46a9-9983-769ec8187f99-host-slash\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802476    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysctl-d\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-systemd\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802630    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-serviceca\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/446f6070-adf6-4559-b8e1-d0e7fc6fff57-cni-binary-copy\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdj5h\" (UniqueName: \"kubernetes.io/projected/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-kube-api-access-cdj5h\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-ovnkube-config\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-env-overrides\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-socket-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-kubelet\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2e694d6-f4a6-48bf-a27c-219fcd70e25c-konnectivity-ca\") pod \"konnectivity-agent-qpnz2\" (UID: \"e2e694d6-f4a6-48bf-a27c-219fcd70e25c\") " pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802831    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-ovnkube-script-lib\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-hostroot\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysconfig\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-var-lib-kubelet\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-lib-modules\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysctl-conf\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802900    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-sysconfig\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-log-socket\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-cni-bin\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.806762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802951    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-kubelet-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803004    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-etc-kubernetes\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-hosts-file\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/03b319fd-30b2-46a9-9983-769ec8187f99-iptables-alerter-script\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.802472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-run-ovn-kubernetes\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-log-socket\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-run-systemd\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803170    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-conf-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-system-cni-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-var-lib-openvswitch\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803236    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-netns\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-run-netns\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803267    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b75189cc-3291-4d83-85d9-c5590098fea5-ovn-node-metrics-cert\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-system-cni-dir\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-cni-bin\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-cni-multus\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-host\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-run-netns\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803436    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5mk\" (UniqueName: \"kubernetes.io/projected/b75189cc-3291-4d83-85d9-c5590098fea5-kube-api-access-kf5mk\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803452    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/446f6070-adf6-4559-b8e1-d0e7fc6fff57-multus-daemon-config\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-cnibin\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmbd4\" (UniqueName: \"kubernetes.io/projected/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-kube-api-access-kmbd4\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-system-cni-dir\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-cni-multus\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803608    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-lib-modules\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803617    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-host\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803680    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-etc-selinux\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b75189cc-3291-4d83-85d9-c5590098fea5-host-run-netns\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803737    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-registration-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-env-overrides\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.803796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-cni-bin\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.807998 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.804270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/446f6070-adf6-4559-b8e1-d0e7fc6fff57-cni-binary-copy\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.804341    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-ovnkube-config\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.804388    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-run\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.804425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-host-var-lib-kubelet\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.804440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b75189cc-3291-4d83-85d9-c5590098fea5-ovnkube-script-lib\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.804465    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/446f6070-adf6-4559-b8e1-d0e7fc6fff57-cnibin\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.804563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eec10c29-003e-489a-af40-bdb5844bfc14-socket-dir\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.806530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-tmp\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.806543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-etc-tuned\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.806918    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2e694d6-f4a6-48bf-a27c-219fcd70e25c-agent-certs\") pod \"konnectivity-agent-qpnz2\" (UID: \"e2e694d6-f4a6-48bf-a27c-219fcd70e25c\") " pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:32.808568 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.807989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b75189cc-3291-4d83-85d9-c5590098fea5-ovn-node-metrics-cert\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.812700 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.812675    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:28:32.812700 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.812703    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:28:32.812881 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.812713    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bhlsq for pod openshift-network-diagnostics/network-check-target-4gnks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:32.812881 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.812784    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq podName:ebfaa30b-0d15-46e0-91ed-b59c86704f8f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:33.312754249 +0000 UTC m=+2.993487272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bhlsq" (UniqueName: "kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq") pod "network-check-target-4gnks" (UID: "ebfaa30b-0d15-46e0-91ed-b59c86704f8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:32.818755 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.818727    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swpkl\" (UniqueName: \"kubernetes.io/projected/446f6070-adf6-4559-b8e1-d0e7fc6fff57-kube-api-access-swpkl\") pod \"multus-7xjsv\" (UID: \"446f6070-adf6-4559-b8e1-d0e7fc6fff57\") " pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:32.818869 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.818859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxfs\" (UniqueName: \"kubernetes.io/projected/8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6-kube-api-access-qgxfs\") pod \"tuned-sb2l5\" (UID: \"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6\") " pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:32.819089 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.819016    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5mk\" (UniqueName: \"kubernetes.io/projected/b75189cc-3291-4d83-85d9-c5590098fea5-kube-api-access-kf5mk\") pod \"ovnkube-node-xxrcb\" (UID: \"b75189cc-3291-4d83-85d9-c5590098fea5\") " pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:32.819346 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.819325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kmd\" (UniqueName: \"kubernetes.io/projected/eec10c29-003e-489a-af40-bdb5844bfc14-kube-api-access-78kmd\") pod \"aws-ebs-csi-driver-node-k2p8t\" (UID: \"eec10c29-003e-489a-af40-bdb5844bfc14\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:32.819646 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.819628    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdj5h\" (UniqueName: \"kubernetes.io/projected/ad9a4e2d-73bb-4e36-84f2-dba812e2812f-kube-api-access-cdj5h\") pod \"node-ca-w8jgr\" (UID: \"ad9a4e2d-73bb-4e36-84f2-dba812e2812f\") " pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:32.820516 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.820495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcs4k\" (UniqueName: \"kubernetes.io/projected/03b319fd-30b2-46a9-9983-769ec8187f99-kube-api-access-fcs4k\") pod \"iptables-alerter-6hj7x\" (UID: \"03b319fd-30b2-46a9-9983-769ec8187f99\") " pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:32.868147 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.867963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal" event={"ID":"d2721bad79031ea0bab955dc2fb6d89c","Type":"ContainerStarted","Data":"ffd431b805b49a2d0ea84e18ad5d60b5dcdf43949d628b6238b09fda645ca828"}
Jul 22 02:28:32.869718 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.869681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal" event={"ID":"bc224443957a17fe5223c8975763ab96","Type":"ContainerStarted","Data":"fe0cbc062b0099cf26037aa3414048db2f0099a7a08fce375d789cd2c05a5f74"}
Jul 22 02:28:32.904254 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-system-cni-dir\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904254 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-cnibin\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904254 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmbd4\" (UniqueName: \"kubernetes.io/projected/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-kube-api-access-kmbd4\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.904254 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904303    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-kubelet-config\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-cnibin\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-system-cni-dir\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-kubelet-config\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98d6\" (UniqueName: \"kubernetes.io/projected/f5aed715-01ac-4ee0-a911-aae835161a19-kube-api-access-t98d6\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-dbus\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-os-release\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-cni-binary-copy\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-tmp-dir\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904626 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.904596    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtcnp\" (UniqueName: \"kubernetes.io/projected/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-kube-api-access-wtcnp\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.904660    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret podName:cdcb05cc-be7e-40e0-943d-e84bc42a16dc nodeName:}" failed. No retries permitted until 2026-07-22 02:28:33.404641124 +0000 UTC m=+3.085374151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret") pod "global-pull-secret-syncer-6grr5" (UID: "cdcb05cc-be7e-40e0-943d-e84bc42a16dc") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904663    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-os-release\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-hosts-file\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-dbus\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:32.904976 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.904924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5aed715-01ac-4ee0-a911-aae835161a19-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.905410 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.904997    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:32.905410 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:32.905070    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs podName:b1b9f796-93fd-4c18-a461-4894a9ce9d0c nodeName:}" failed. No retries permitted until 2026-07-22 02:28:33.405056024 +0000 UTC m=+3.085789049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs") pod "network-metrics-daemon-5t4kv" (UID: "b1b9f796-93fd-4c18-a461-4894a9ce9d0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:32.905410 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.905125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-hosts-file\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.905410 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.905350    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-cni-binary-copy\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.905410 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.905382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.905681 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.905470    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f5aed715-01ac-4ee0-a911-aae835161a19-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.905681 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.905647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-tmp-dir\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.916642 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.916601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmbd4\" (UniqueName: \"kubernetes.io/projected/a4e8bb86-f9d8-440a-a1eb-0f59bb913304-kube-api-access-kmbd4\") pod \"node-resolver-t2hdl\" (UID: \"a4e8bb86-f9d8-440a-a1eb-0f59bb913304\") " pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:32.917716 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.917694    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98d6\" (UniqueName: \"kubernetes.io/projected/f5aed715-01ac-4ee0-a911-aae835161a19-kube-api-access-t98d6\") pod \"multus-additional-cni-plugins-zfxnf\" (UID: \"f5aed715-01ac-4ee0-a911-aae835161a19\") " pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:32.918337 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.918318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtcnp\" (UniqueName: \"kubernetes.io/projected/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-kube-api-access-wtcnp\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:32.983177 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:32.983138    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:33.005743 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.005709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6hj7x"
Jul 22 02:28:33.013677 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.013650    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t"
Jul 22 02:28:33.020619 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.020595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7xjsv"
Jul 22 02:28:33.027581 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.027550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sb2l5"
Jul 22 02:28:33.036446 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.036425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w8jgr"
Jul 22 02:28:33.044543 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.044519    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:33.051429 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.051409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t2hdl"
Jul 22 02:28:33.057339 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.057318    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zfxnf"
Jul 22 02:28:33.407731 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.407699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:33.407897 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.407743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:33.407897 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.407773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:33.407897 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.407850    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:33.408014 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.407905    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs podName:b1b9f796-93fd-4c18-a461-4894a9ce9d0c nodeName:}" failed. No retries permitted until 2026-07-22 02:28:34.407891083 +0000 UTC m=+4.088624120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs") pod "network-metrics-daemon-5t4kv" (UID: "b1b9f796-93fd-4c18-a461-4894a9ce9d0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:33.408014 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.407910    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:28:33.408014 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.407850    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:33.408014 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.407942    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:28:33.408014 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.407955    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bhlsq for pod openshift-network-diagnostics/network-check-target-4gnks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:33.408014 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.407970    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret podName:cdcb05cc-be7e-40e0-943d-e84bc42a16dc nodeName:}" failed. No retries permitted until 2026-07-22 02:28:34.40795958 +0000 UTC m=+4.088692603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret") pod "global-pull-secret-syncer-6grr5" (UID: "cdcb05cc-be7e-40e0-943d-e84bc42a16dc") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:33.408228 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.408079    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq podName:ebfaa30b-0d15-46e0-91ed-b59c86704f8f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:34.408061041 +0000 UTC m=+4.088794069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bhlsq" (UniqueName: "kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq") pod "network-check-target-4gnks" (UID: "ebfaa30b-0d15-46e0-91ed-b59c86704f8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:33.730937 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.730903    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:23:32 +0000 UTC" deadline="2027-12-18 16:38:58.058866845 +0000 UTC"
Jul 22 02:28:33.731298 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.730942    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12350h10m24.327931454s"
Jul 22 02:28:33.737227 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:33.737194    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9a4e2d_73bb_4e36_84f2_dba812e2812f.slice/crio-3718ea5a6370389631ce1b1930796fdd3df1d5dc544cf3d75c42aa3d6dc2a4fc WatchSource:0}: Error finding container 3718ea5a6370389631ce1b1930796fdd3df1d5dc544cf3d75c42aa3d6dc2a4fc: Status 404 returned error can't find the container with id 3718ea5a6370389631ce1b1930796fdd3df1d5dc544cf3d75c42aa3d6dc2a4fc
Jul 22 02:28:33.739548 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:33.739522    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acc0ab1_e4ea_4b74_a9e2_f1581f7f4fc6.slice/crio-7287dc24dc2b62afdf16d04fef789c8640aceb9a14467cf1c8258870608797ee WatchSource:0}: Error finding container 7287dc24dc2b62afdf16d04fef789c8640aceb9a14467cf1c8258870608797ee: Status 404 returned error can't find the container with id 7287dc24dc2b62afdf16d04fef789c8640aceb9a14467cf1c8258870608797ee
Jul 22 02:28:33.740331 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:33.740303    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b319fd_30b2_46a9_9983_769ec8187f99.slice/crio-97d2e203efda33d248e7b6e435de8003c43a6df0f810e9525b5e0e241e9ff2a1 WatchSource:0}: Error finding container 97d2e203efda33d248e7b6e435de8003c43a6df0f810e9525b5e0e241e9ff2a1: Status 404 returned error can't find the container with id 97d2e203efda33d248e7b6e435de8003c43a6df0f810e9525b5e0e241e9ff2a1
Jul 22 02:28:33.741080 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:33.741022    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5aed715_01ac_4ee0_a911_aae835161a19.slice/crio-9ff136e67dfae42927c3c9a84720e55d4cc463469fb8f4277b096740fafdffaf WatchSource:0}: Error finding container 9ff136e67dfae42927c3c9a84720e55d4cc463469fb8f4277b096740fafdffaf: Status 404 returned error can't find the container with id 9ff136e67dfae42927c3c9a84720e55d4cc463469fb8f4277b096740fafdffaf
Jul 22 02:28:33.741980 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:33.741956    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446f6070_adf6_4559_b8e1_d0e7fc6fff57.slice/crio-ed4d0ebd07c94837c6335c543b910f105d1c1cbb0e6a5752ff6ead41ac1dbf1a WatchSource:0}: Error finding container ed4d0ebd07c94837c6335c543b910f105d1c1cbb0e6a5752ff6ead41ac1dbf1a: Status 404 returned error can't find the container with id ed4d0ebd07c94837c6335c543b910f105d1c1cbb0e6a5752ff6ead41ac1dbf1a
Jul 22 02:28:33.743375 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:28:33.743350    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e694d6_f4a6_48bf_a27c_219fcd70e25c.slice/crio-a1d22622488aa81baff3f9d32500b474940408c3d91a4056a31425d0b8befa86 WatchSource:0}: Error finding container a1d22622488aa81baff3f9d32500b474940408c3d91a4056a31425d0b8befa86: Status 404 returned error can't find the container with id a1d22622488aa81baff3f9d32500b474940408c3d91a4056a31425d0b8befa86
Jul 22 02:28:33.863179 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.862962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:33.863325 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:33.863259    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:33.872456 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.872422    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w8jgr" event={"ID":"ad9a4e2d-73bb-4e36-84f2-dba812e2812f","Type":"ContainerStarted","Data":"3718ea5a6370389631ce1b1930796fdd3df1d5dc544cf3d75c42aa3d6dc2a4fc"}
Jul 22 02:28:33.874628 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.874601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal" event={"ID":"bc224443957a17fe5223c8975763ab96","Type":"ContainerStarted","Data":"6fc039a6e2a9c2d40d64bdc6eae3271b94ec45f76fdfc829c788bd65827b8d93"}
Jul 22 02:28:33.879088 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.878973    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qpnz2" event={"ID":"e2e694d6-f4a6-48bf-a27c-219fcd70e25c","Type":"ContainerStarted","Data":"a1d22622488aa81baff3f9d32500b474940408c3d91a4056a31425d0b8befa86"}
Jul 22 02:28:33.881088 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.881059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerStarted","Data":"9ff136e67dfae42927c3c9a84720e55d4cc463469fb8f4277b096740fafdffaf"}
Jul 22 02:28:33.882570 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.882544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xjsv" event={"ID":"446f6070-adf6-4559-b8e1-d0e7fc6fff57","Type":"ContainerStarted","Data":"ed4d0ebd07c94837c6335c543b910f105d1c1cbb0e6a5752ff6ead41ac1dbf1a"}
Jul 22 02:28:33.883759 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.883738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"4f65fa768ce1214d941714909ffcef353c89c25856c34ae1595b4b203ae81206"}
Jul 22 02:28:33.885144 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.885122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t" event={"ID":"eec10c29-003e-489a-af40-bdb5844bfc14","Type":"ContainerStarted","Data":"c5ca18273ef2ff793b50d56ce9919b1cc250f8db62f3f75b30af42c5ef5fc9ac"}
Jul 22 02:28:33.886906 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.886888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hj7x" event={"ID":"03b319fd-30b2-46a9-9983-769ec8187f99","Type":"ContainerStarted","Data":"97d2e203efda33d248e7b6e435de8003c43a6df0f810e9525b5e0e241e9ff2a1"}
Jul 22 02:28:33.888110 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.888092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sb2l5" event={"ID":"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6","Type":"ContainerStarted","Data":"7287dc24dc2b62afdf16d04fef789c8640aceb9a14467cf1c8258870608797ee"}
Jul 22 02:28:33.888981 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:33.888956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t2hdl" event={"ID":"a4e8bb86-f9d8-440a-a1eb-0f59bb913304","Type":"ContainerStarted","Data":"bb51b55dd02663f8f39e47e985cddf36f850b0dbd96cc2c274d959d5092af2bc"}
Jul 22 02:28:34.415885 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.415843    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:34.416100 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.415905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:34.416100 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.415934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:34.416100 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.416090    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:34.416265 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.416150    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret podName:cdcb05cc-be7e-40e0-943d-e84bc42a16dc nodeName:}" failed. No retries permitted until 2026-07-22 02:28:36.416130648 +0000 UTC m=+6.096863692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret") pod "global-pull-secret-syncer-6grr5" (UID: "cdcb05cc-be7e-40e0-943d-e84bc42a16dc") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:34.416994 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.416671    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:34.416994 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.416720    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs podName:b1b9f796-93fd-4c18-a461-4894a9ce9d0c nodeName:}" failed. No retries permitted until 2026-07-22 02:28:36.416704597 +0000 UTC m=+6.097437638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs") pod "network-metrics-daemon-5t4kv" (UID: "b1b9f796-93fd-4c18-a461-4894a9ce9d0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:34.416994 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.416793    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:28:34.416994 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.416808    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:28:34.416994 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.416820    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bhlsq for pod openshift-network-diagnostics/network-check-target-4gnks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:34.416994 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.416853    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq podName:ebfaa30b-0d15-46e0-91ed-b59c86704f8f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:36.4168418 +0000 UTC m=+6.097574838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bhlsq" (UniqueName: "kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq") pod "network-check-target-4gnks" (UID: "ebfaa30b-0d15-46e0-91ed-b59c86704f8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:34.733679 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.733613    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 02:23:32 +0000 UTC" deadline="2028-03-20 20:15:32.683506462 +0000 UTC"
Jul 22 02:28:34.733679 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.733677    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14585h46m57.949833008s"
Jul 22 02:28:34.864189 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.864154    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:34.864369 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.864271    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:34.864686 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.864663    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:34.864790 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:34.864771    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:34.907590 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.907549    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2721bad79031ea0bab955dc2fb6d89c" containerID="24053d3200d0a9deb22bdeffe7670d31e7057310ab44f6a570652ab59960345a" exitCode=0
Jul 22 02:28:34.908745 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.908465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal" event={"ID":"d2721bad79031ea0bab955dc2fb6d89c","Type":"ContainerDied","Data":"24053d3200d0a9deb22bdeffe7670d31e7057310ab44f6a570652ab59960345a"}
Jul 22 02:28:34.926607 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:34.926555    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-30.ec2.internal" podStartSLOduration=2.926533526 podStartE2EDuration="2.926533526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:33.889442529 +0000 UTC m=+3.570175577" watchObservedRunningTime="2026-07-22 02:28:34.926533526 +0000 UTC m=+4.607266570"
Jul 22 02:28:35.863691 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:35.863203    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:35.863691 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:35.863340    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:35.924754 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:35.924717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal" event={"ID":"d2721bad79031ea0bab955dc2fb6d89c","Type":"ContainerStarted","Data":"d2492d2896cf3f976fbb31a6dfc3d9da80c3a7711eee3359185c05722cfe426b"}
Jul 22 02:28:36.435256 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:36.435211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:36.435444 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:36.435286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:36.435444 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:36.435319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:36.435444 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.435393    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:36.435595 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.435457    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:36.435595 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.435466    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs podName:b1b9f796-93fd-4c18-a461-4894a9ce9d0c nodeName:}" failed. No retries permitted until 2026-07-22 02:28:40.435447013 +0000 UTC m=+10.116180051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs") pod "network-metrics-daemon-5t4kv" (UID: "b1b9f796-93fd-4c18-a461-4894a9ce9d0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:36.435595 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.435508    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret podName:cdcb05cc-be7e-40e0-943d-e84bc42a16dc nodeName:}" failed. No retries permitted until 2026-07-22 02:28:40.435493035 +0000 UTC m=+10.116226075 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret") pod "global-pull-secret-syncer-6grr5" (UID: "cdcb05cc-be7e-40e0-943d-e84bc42a16dc") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:36.435595 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.435582    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:28:36.435794 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.435601    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:28:36.435794 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.435614    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bhlsq for pod openshift-network-diagnostics/network-check-target-4gnks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:36.435794 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.435655    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq podName:ebfaa30b-0d15-46e0-91ed-b59c86704f8f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:40.435641459 +0000 UTC m=+10.116374497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bhlsq" (UniqueName: "kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq") pod "network-check-target-4gnks" (UID: "ebfaa30b-0d15-46e0-91ed-b59c86704f8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:36.868059 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:36.865431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:36.868059 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.865577    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:36.868059 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:36.865931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:36.868059 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:36.866021    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:37.863137 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:37.863102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:37.863339 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:37.863236    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:38.866267 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:38.865557    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:38.866267 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:38.865682    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:38.866267 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:38.866088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:38.866267 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:38.866201    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:39.863720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:39.863687    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:39.863888 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:39.863819    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:40.471649 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:40.471590    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:40.472065 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:40.471662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:40.472065 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:40.471682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:40.472065 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.471778    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:40.472065 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.471824    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret podName:cdcb05cc-be7e-40e0-943d-e84bc42a16dc nodeName:}" failed. No retries permitted until 2026-07-22 02:28:48.471810593 +0000 UTC m=+18.152543630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret") pod "global-pull-secret-syncer-6grr5" (UID: "cdcb05cc-be7e-40e0-943d-e84bc42a16dc") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:40.472217 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.472132    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:40.472217 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.472167    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs podName:b1b9f796-93fd-4c18-a461-4894a9ce9d0c nodeName:}" failed. No retries permitted until 2026-07-22 02:28:48.472155896 +0000 UTC m=+18.152888933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs") pod "network-metrics-daemon-5t4kv" (UID: "b1b9f796-93fd-4c18-a461-4894a9ce9d0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:40.472293 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.472217    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:28:40.472293 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.472227    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:28:40.472293 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.472234    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bhlsq for pod openshift-network-diagnostics/network-check-target-4gnks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:40.472293 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.472256    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq podName:ebfaa30b-0d15-46e0-91ed-b59c86704f8f nodeName:}" failed. No retries permitted until 2026-07-22 02:28:48.47224937 +0000 UTC m=+18.152982407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bhlsq" (UniqueName: "kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq") pod "network-check-target-4gnks" (UID: "ebfaa30b-0d15-46e0-91ed-b59c86704f8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:40.867743 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:40.867130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:40.867743 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.867245    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:40.867743 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:40.867592    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:40.867743 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:40.867700    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:41.863768 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:41.863249    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:41.863768 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:41.863387    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:42.866020 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:42.865460    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:42.866020 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:42.865577    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:42.866020 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:42.865902    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:42.866020 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:42.865978    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:43.863421 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:43.863384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:43.863601 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:43.863515    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:44.863387 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:44.863322    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:44.863842 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:44.863466    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:44.863842 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:44.863526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:44.863842 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:44.863630    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:45.863720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:45.863686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:45.864211 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:45.863809    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:46.863231 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:46.863187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:46.863392 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:46.863354    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:46.863461 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:46.863415    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:46.863583 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:46.863548    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:47.863486 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:47.863453    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:47.864050 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:47.863566    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:48.533023 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:48.532984    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:48.533232 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:48.533047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:48.533232 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:48.533066    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:48.533232 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.533184    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:48.533401 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.533241    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret podName:cdcb05cc-be7e-40e0-943d-e84bc42a16dc nodeName:}" failed. No retries permitted until 2026-07-22 02:29:04.533222952 +0000 UTC m=+34.213955990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret") pod "global-pull-secret-syncer-6grr5" (UID: "cdcb05cc-be7e-40e0-943d-e84bc42a16dc") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:28:48.533459 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.533400    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:48.533459 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.533433    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:28:48.533459 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.533451    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs podName:b1b9f796-93fd-4c18-a461-4894a9ce9d0c nodeName:}" failed. No retries permitted until 2026-07-22 02:29:04.533437471 +0000 UTC m=+34.214170495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs") pod "network-metrics-daemon-5t4kv" (UID: "b1b9f796-93fd-4c18-a461-4894a9ce9d0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:28:48.533598 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.533467    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:28:48.533598 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.533479    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bhlsq for pod openshift-network-diagnostics/network-check-target-4gnks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:48.533598 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.533537    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq podName:ebfaa30b-0d15-46e0-91ed-b59c86704f8f nodeName:}" failed. No retries permitted until 2026-07-22 02:29:04.533519931 +0000 UTC m=+34.214252955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bhlsq" (UniqueName: "kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq") pod "network-check-target-4gnks" (UID: "ebfaa30b-0d15-46e0-91ed-b59c86704f8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:28:48.863806 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:48.863731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:48.864237 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:48.863732    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:48.864237 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.863906    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:48.864237 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:48.864056    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:49.863420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:49.863387    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:49.863602 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:49.863504    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:50.864074 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:50.864020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:50.864483 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:50.864168    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:50.864483 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:50.864227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:50.864483 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:50.864328    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:51.863693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:51.863658    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:51.863865 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:51.863793    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:52.863145 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.862685    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:52.863145 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:52.862988    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:52.863145 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.863000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:52.863145 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:52.863128    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:52.960308 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.960267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qpnz2" event={"ID":"e2e694d6-f4a6-48bf-a27c-219fcd70e25c","Type":"ContainerStarted","Data":"1438e00b715474f7801bd2dda4a29a9698de8d175a035ab2faff3768a814c12d"}
Jul 22 02:28:52.961884 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.961858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerStarted","Data":"4a459b77a27fe4c21151c0b77b1367c16b5c89f70b87b533dee9deff590b746c"}
Jul 22 02:28:52.966768 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.966743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7xjsv" event={"ID":"446f6070-adf6-4559-b8e1-d0e7fc6fff57","Type":"ContainerStarted","Data":"1f82138b681f1c0faaaa9a31315c3d218aa0ae003c976058fa607463f0fa4fe9"}
Jul 22 02:28:52.968169 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.968146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"6c8b9b647ca2b69ba18255e38a91c4d441836591d25e486767eeb8ac8511a822"}
Jul 22 02:28:52.969551 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.969528    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t" event={"ID":"eec10c29-003e-489a-af40-bdb5844bfc14","Type":"ContainerStarted","Data":"fde836bdd058e4a8372d416362277fb8c237083d40c3a6742e2e0165a8b10109"}
Jul 22 02:28:52.970955 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.970933    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sb2l5" event={"ID":"8acc0ab1-e4ea-4b74-a9e2-f1581f7f4fc6","Type":"ContainerStarted","Data":"4a8a892626a12e606ca2a14f4d984ba60c4c72274ebf2465bdfbda1a536ce281"}
Jul 22 02:28:52.972379 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.972354    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t2hdl" event={"ID":"a4e8bb86-f9d8-440a-a1eb-0f59bb913304","Type":"ContainerStarted","Data":"45fbfdc0581d169215f87ba8febfc685b0fe81beb660138f7088ecccbc3ed162"}
Jul 22 02:28:52.973638 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.973617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w8jgr" event={"ID":"ad9a4e2d-73bb-4e36-84f2-dba812e2812f","Type":"ContainerStarted","Data":"afc687501acbcbd8daebb0495913f1a4f9dc3b0cc66a00d5c64b16f01e37b0d4"}
Jul 22 02:28:52.976954 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.976918    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-30.ec2.internal" podStartSLOduration=20.976906509 podStartE2EDuration="20.976906509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:35.945088754 +0000 UTC m=+5.625821800" watchObservedRunningTime="2026-07-22 02:28:52.976906509 +0000 UTC m=+22.657639554"
Jul 22 02:28:52.977170 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.977149    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qpnz2" podStartSLOduration=8.805619132 podStartE2EDuration="22.977142539s" totalImagesPullingTime="14.171523407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:52.976611361 +0000 UTC m=+22.657344409" watchObservedRunningTime="2026-07-22 02:28:52.977142539 +0000 UTC m=+22.657875585"
Jul 22 02:28:52.993767 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:52.993726    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7xjsv" podStartSLOduration=4.046915366 podStartE2EDuration="22.993711374s" totalImagesPullingTime="18.946796008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:52.993324856 +0000 UTC m=+22.674057905" watchObservedRunningTime="2026-07-22 02:28:52.993711374 +0000 UTC m=+22.674444420"
Jul 22 02:28:53.011950 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.011905    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sb2l5" podStartSLOduration=4.077512654 podStartE2EDuration="23.011888998s" totalImagesPullingTime="18.934376344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:53.011545213 +0000 UTC m=+22.692278259" watchObservedRunningTime="2026-07-22 02:28:53.011888998 +0000 UTC m=+22.692622075"
Jul 22 02:28:53.030065 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.029990    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t2hdl" podStartSLOduration=4.147941089 podStartE2EDuration="23.029973784s" totalImagesPullingTime="18.882032695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:53.029538975 +0000 UTC m=+22.710272022" watchObservedRunningTime="2026-07-22 02:28:53.029973784 +0000 UTC m=+22.710706830"
Jul 22 02:28:53.044058 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.043895    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w8jgr" podStartSLOduration=4.110158391 podStartE2EDuration="23.043871702s" totalImagesPullingTime="18.933713311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:53.043267602 +0000 UTC m=+22.724000669" watchObservedRunningTime="2026-07-22 02:28:53.043871702 +0000 UTC m=+22.724604749"
Jul 22 02:28:53.852240 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.852079    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 22 02:28:53.863347 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.863326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:53.864096 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:53.863449    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:53.977340 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.977261    2583 generic.go:356] "Generic (PLEG): container finished" podID="f5aed715-01ac-4ee0-a911-aae835161a19" containerID="4a459b77a27fe4c21151c0b77b1367c16b5c89f70b87b533dee9deff590b746c" exitCode=0
Jul 22 02:28:53.977497 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.977348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerDied","Data":"4a459b77a27fe4c21151c0b77b1367c16b5c89f70b87b533dee9deff590b746c"}
Jul 22 02:28:53.980449 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.980429    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:28:53.980824 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.980806    2583 generic.go:356] "Generic (PLEG): container finished" podID="b75189cc-3291-4d83-85d9-c5590098fea5" containerID="f144ea1f7bbb7c951c369e01847811c5e4238476800a2a34cb0c496002c43b36" exitCode=1
Jul 22 02:28:53.980894 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.980881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"d6f9cddd7300866a22702f4242e3da346ee7dc46885d1afbad5dafff5d4c343e"}
Jul 22 02:28:53.980977 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.980958    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"aca779932155328925318e47942662d38d13e0980595ee3cce1caa359ba7fea5"}
Jul 22 02:28:53.981082 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.981017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"923674ef8777ba79aabdfa5e8eaeeed25296261bda2d767c6345339a5124d55d"}
Jul 22 02:28:53.981155 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.981087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"8db1222b5b17eba1227cc5dbb392525f8cb9fa9a59914aab9166a57d3cc9ba4d"}
Jul 22 02:28:53.981155 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.981147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerDied","Data":"f144ea1f7bbb7c951c369e01847811c5e4238476800a2a34cb0c496002c43b36"}
Jul 22 02:28:53.982683 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:53.982624    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t" event={"ID":"eec10c29-003e-489a-af40-bdb5844bfc14","Type":"ContainerStarted","Data":"64b9ffd53270b4055de66a2e54c58614a0029de025cbcccaf503bddf0b0e4b85"}
Jul 22 02:28:54.756114 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:54.755970    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T02:28:53.852231956Z","UUID":"ddc38d7d-da54-45e9-8e74-92a6e6825ebe","Handler":null,"Name":"","Endpoint":""}
Jul 22 02:28:54.758017 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:54.757988    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 22 02:28:54.758168 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:54.758063    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 22 02:28:54.863184 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:54.863153    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:54.863318 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:54.863296    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:54.863377 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:54.863357    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:54.863671 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:54.863475    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:54.987710 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:54.987678    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t" event={"ID":"eec10c29-003e-489a-af40-bdb5844bfc14","Type":"ContainerStarted","Data":"c75c8035b334164221bad730d5fbb0f96be323e66547df420e2a2d7eb93e1a7d"}
Jul 22 02:28:54.989586 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:54.989549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6hj7x" event={"ID":"03b319fd-30b2-46a9-9983-769ec8187f99","Type":"ContainerStarted","Data":"00aafae36c5b1719881bb9acbb504156ac6a6c100cb50dc19cdc1d074c0029c3"}
Jul 22 02:28:55.007301 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.007192    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-k2p8t" podStartSLOduration=4.246398967 podStartE2EDuration="25.007171729s" totalImagesPullingTime="20.760772762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:55.006245535 +0000 UTC m=+24.686978579" watchObservedRunningTime="2026-07-22 02:28:55.007171729 +0000 UTC m=+24.687904782"
Jul 22 02:28:55.023187 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.023138    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6hj7x" podStartSLOduration=6.09128856 podStartE2EDuration="25.023121557s" totalImagesPullingTime="18.931832997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:55.023065971 +0000 UTC m=+24.703799018" watchObservedRunningTime="2026-07-22 02:28:55.023121557 +0000 UTC m=+24.703854607"
Jul 22 02:28:55.842427 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.842192    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:55.842901 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.842879    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:55.862932 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.862903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:55.863129 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:55.863011    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:55.995518 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.995487    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:28:55.995941 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.995898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"abb108257db4d80abe50673ced0c7949fa23832c7eefe3ae3dff3166c34ce133"}
Jul 22 02:28:55.996258 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.996228    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:55.996638 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:55.996622    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qpnz2"
Jul 22 02:28:56.863379 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:56.863349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:56.863583 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:56.863469    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:56.863583 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:56.863527    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:56.863727 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:56.863632    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:57.863700 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:57.863501    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:57.863700 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:57.863624    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:28:58.008888 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.008717    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:28:58.009449 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.009308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"3da33f2ac216aa2f116d6c62eec76f338b45b9cb6fbc0da52b3ab83665104975"}
Jul 22 02:28:58.009832 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.009812    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:58.009901 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.009844    2583 scope.go:117] "RemoveContainer" containerID="f144ea1f7bbb7c951c369e01847811c5e4238476800a2a34cb0c496002c43b36"
Jul 22 02:28:58.009901 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.009890    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:58.010011 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.009932    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:58.012125 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.012097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerStarted","Data":"90f3a433534e6bdb8f2de730aef90f970aa5ac676d7c32f4244ab00ce46537be"}
Jul 22 02:28:58.029281 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.029257    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:58.029978 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.029867    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:28:58.862707 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.862682    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:28:58.862935 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:58.862798    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:28:58.862935 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:58.862832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:28:58.863074 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:58.862967    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:28:59.015528 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:59.015500    2583 generic.go:356] "Generic (PLEG): container finished" podID="f5aed715-01ac-4ee0-a911-aae835161a19" containerID="90f3a433534e6bdb8f2de730aef90f970aa5ac676d7c32f4244ab00ce46537be" exitCode=0
Jul 22 02:28:59.016045 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:59.015566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerDied","Data":"90f3a433534e6bdb8f2de730aef90f970aa5ac676d7c32f4244ab00ce46537be"}
Jul 22 02:28:59.019322 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:59.019304    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:28:59.019709 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:59.019688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" event={"ID":"b75189cc-3291-4d83-85d9-c5590098fea5","Type":"ContainerStarted","Data":"936acb56ff031dc8c9fd993dbead496021b4fbdf43036524507e73ef0e019f0e"}
Jul 22 02:28:59.108680 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:59.108629    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb" podStartSLOduration=10.109723558 podStartE2EDuration="29.108614082s" totalImagesPullingTime="18.998890524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:28:59.107059687 +0000 UTC m=+28.787792733" watchObservedRunningTime="2026-07-22 02:28:59.108614082 +0000 UTC m=+28.789347127"
Jul 22 02:28:59.863086 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:28:59.862982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:28:59.863234 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:28:59.863108    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:29:00.023586 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:00.023545    2583 generic.go:356] "Generic (PLEG): container finished" podID="f5aed715-01ac-4ee0-a911-aae835161a19" containerID="8e82d823810009a4ca8862fa4dbc2287bd29dc002538908351d53633d3befce6" exitCode=0
Jul 22 02:29:00.023975 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:00.023614    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerDied","Data":"8e82d823810009a4ca8862fa4dbc2287bd29dc002538908351d53633d3befce6"}
Jul 22 02:29:00.863907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:00.863815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:00.864065 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:00.863932    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:29:00.864065 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:00.863990    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:00.864139 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:00.864105    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:29:01.028535 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:01.028501    2583 generic.go:356] "Generic (PLEG): container finished" podID="f5aed715-01ac-4ee0-a911-aae835161a19" containerID="1248bb406b96376f32801aa334bb9bde48b7a54ca5f30ffee97e499067afc99a" exitCode=0
Jul 22 02:29:01.028902 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:01.028556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerDied","Data":"1248bb406b96376f32801aa334bb9bde48b7a54ca5f30ffee97e499067afc99a"}
Jul 22 02:29:01.863227 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:01.862789    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:01.863227 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:01.862902    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:29:02.864241 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:02.863647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:02.864241 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:02.863782    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:29:02.864241 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:02.864101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:02.864241 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:02.864194    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:29:03.863203 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:03.863165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:03.863458 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:03.863312    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:29:04.542090 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:04.542053    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:04.542109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:04.542137    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.542232    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.542236    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.542288    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret podName:cdcb05cc-be7e-40e0-943d-e84bc42a16dc nodeName:}" failed. No retries permitted until 2026-07-22 02:29:36.542272108 +0000 UTC m=+66.223005145 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret") pod "global-pull-secret-syncer-6grr5" (UID: "cdcb05cc-be7e-40e0-943d-e84bc42a16dc") : object "kube-system"/"original-pull-secret" not registered
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.542308    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs podName:b1b9f796-93fd-4c18-a461-4894a9ce9d0c nodeName:}" failed. No retries permitted until 2026-07-22 02:29:36.542297171 +0000 UTC m=+66.223030195 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs") pod "network-metrics-daemon-5t4kv" (UID: "b1b9f796-93fd-4c18-a461-4894a9ce9d0c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.542355    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.542369    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.542378    2583 projected.go:196] Error preparing data for projected volume kube-api-access-bhlsq for pod openshift-network-diagnostics/network-check-target-4gnks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:29:04.542544 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.542419    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq podName:ebfaa30b-0d15-46e0-91ed-b59c86704f8f nodeName:}" failed. No retries permitted until 2026-07-22 02:29:36.54240473 +0000 UTC m=+66.223137771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bhlsq" (UniqueName: "kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq") pod "network-check-target-4gnks" (UID: "ebfaa30b-0d15-46e0-91ed-b59c86704f8f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 02:29:04.863132 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:04.863043    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:04.863302 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.863152    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:29:04.863302 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:04.863189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:04.863302 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:04.863264    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:29:05.862967 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:05.862813    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:05.863339 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:05.863106    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:29:06.043365 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:06.043284    2583 generic.go:356] "Generic (PLEG): container finished" podID="f5aed715-01ac-4ee0-a911-aae835161a19" containerID="cd900203094dd9a69a5aa64c239c31c57f9918a30221db2ec635c54e03273691" exitCode=0
Jul 22 02:29:06.043365 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:06.043333    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerDied","Data":"cd900203094dd9a69a5aa64c239c31c57f9918a30221db2ec635c54e03273691"}
Jul 22 02:29:06.863330 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:06.863302    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:06.863330 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:06.863324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:06.863880 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:06.863410    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:29:06.863880 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:06.863531    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:29:07.048281 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:07.048238    2583 generic.go:356] "Generic (PLEG): container finished" podID="f5aed715-01ac-4ee0-a911-aae835161a19" containerID="5675b88678767caf05d9b50c197c3da9ce0c681c44a481fe052a81ddb2591cba" exitCode=0
Jul 22 02:29:07.048440 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:07.048307    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerDied","Data":"5675b88678767caf05d9b50c197c3da9ce0c681c44a481fe052a81ddb2591cba"}
Jul 22 02:29:07.365220 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:07.365185    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6grr5"]
Jul 22 02:29:07.365372 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:07.365314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:07.365427 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:07.365408    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:29:07.368975 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:07.368867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gnks"]
Jul 22 02:29:07.369145 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:07.369005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:07.369219 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:07.369148    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:29:07.369738 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:07.369535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5t4kv"]
Jul 22 02:29:07.369738 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:07.369637    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:07.369953 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:07.369920    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:29:08.053408 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:08.053369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" event={"ID":"f5aed715-01ac-4ee0-a911-aae835161a19","Type":"ContainerStarted","Data":"db06ce2bf3ddf96e9e22bea24611063b377fdcfeff099677576190da41a35f86"}
Jul 22 02:29:08.078346 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:08.078297    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zfxnf" podStartSLOduration=8.40382358 podStartE2EDuration="37.078280792s" totalImagesPullingTime="28.674457212s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:08.078068841 +0000 UTC m=+37.758801887" watchObservedRunningTime="2026-07-22 02:29:08.078280792 +0000 UTC m=+37.759013859"
Jul 22 02:29:08.863173 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:08.863134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:08.863173 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:08.863161    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:08.863404 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:08.863197    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:08.863404 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:08.863289    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:29:08.863404 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:08.863381    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:29:08.863496 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:08.863458    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:29:10.863475 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:10.863442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:10.863849 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:10.863548    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5t4kv" podUID="b1b9f796-93fd-4c18-a461-4894a9ce9d0c"
Jul 22 02:29:10.863849 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:10.863570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:10.863849 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:10.863596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:10.863849 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:10.863689    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gnks" podUID="ebfaa30b-0d15-46e0-91ed-b59c86704f8f"
Jul 22 02:29:10.863849 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:10.863760    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6grr5" podUID="cdcb05cc-be7e-40e0-943d-e84bc42a16dc"
Jul 22 02:29:11.667490 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.667420    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-30.ec2.internal" event="NodeReady"
Jul 22 02:29:11.667628 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.667551    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 22 02:29:11.750679 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.750640    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-njphq"]
Jul 22 02:29:11.779183 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.779154    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x6fbg"]
Jul 22 02:29:11.779326 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.779272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:11.782011 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.781988    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 22 02:29:11.782162 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.781991    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 22 02:29:11.782303 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.782285    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 22 02:29:11.782831 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.782818    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-btmmd\""
Jul 22 02:29:11.783686 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.783671    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 22 02:29:11.800417 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.800397    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-njphq"]
Jul 22 02:29:11.800531 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.800431    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6fbg"]
Jul 22 02:29:11.800531 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.800448    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vx8b4"]
Jul 22 02:29:11.800531 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.800493    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:11.806573 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.803617    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 22 02:29:11.806573 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.803645    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 22 02:29:11.806573 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.803674    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2lw7w\""
Jul 22 02:29:11.826463 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.826432    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vx8b4"]
Jul 22 02:29:11.826630 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.826560    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vx8b4"
Jul 22 02:29:11.829434 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.829414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vvdpl\""
Jul 22 02:29:11.829555 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.829460    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 22 02:29:11.832647 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.832631    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 22 02:29:11.832741 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.832676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 22 02:29:11.899280 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb1cb847-7b39-496c-ab68-ed6478050b8e-data-volume\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:11.899280 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a283d73-3254-49ee-bd3d-32015aea25e0-config-volume\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:11.899693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1cb847-7b39-496c-ab68-ed6478050b8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:11.899693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a283d73-3254-49ee-bd3d-32015aea25e0-metrics-tls\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:11.899693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1cb847-7b39-496c-ab68-ed6478050b8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:11.899693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb1cb847-7b39-496c-ab68-ed6478050b8e-crio-socket\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:11.899693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899527    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwsq2\" (UniqueName: \"kubernetes.io/projected/eb1cb847-7b39-496c-ab68-ed6478050b8e-kube-api-access-rwsq2\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:11.899693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmj8z\" (UniqueName: \"kubernetes.io/projected/8a283d73-3254-49ee-bd3d-32015aea25e0-kube-api-access-xmj8z\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:11.899693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:11.899581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a283d73-3254-49ee-bd3d-32015aea25e0-tmp-dir\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.000171 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000141    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1cb847-7b39-496c-ab68-ed6478050b8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.000267 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000199    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a283d73-3254-49ee-bd3d-32015aea25e0-metrics-tls\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.000267 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1cb847-7b39-496c-ab68-ed6478050b8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.000267 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb1cb847-7b39-496c-ab68-ed6478050b8e-crio-socket\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.000394 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f191e60-ef7e-4fc1-93da-9081634c8ba4-cert\") pod \"ingress-canary-vx8b4\" (UID: \"1f191e60-ef7e-4fc1-93da-9081634c8ba4\") " pod="openshift-ingress-canary/ingress-canary-vx8b4"
Jul 22 02:29:12.000394 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwsq2\" (UniqueName: \"kubernetes.io/projected/eb1cb847-7b39-496c-ab68-ed6478050b8e-kube-api-access-rwsq2\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.000394 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmj8z\" (UniqueName: \"kubernetes.io/projected/8a283d73-3254-49ee-bd3d-32015aea25e0-kube-api-access-xmj8z\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.000394 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000361    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a283d73-3254-49ee-bd3d-32015aea25e0-tmp-dir\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.000570 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclb8\" (UniqueName: \"kubernetes.io/projected/1f191e60-ef7e-4fc1-93da-9081634c8ba4-kube-api-access-sclb8\") pod \"ingress-canary-vx8b4\" (UID: \"1f191e60-ef7e-4fc1-93da-9081634c8ba4\") " pod="openshift-ingress-canary/ingress-canary-vx8b4"
Jul 22 02:29:12.000570 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb1cb847-7b39-496c-ab68-ed6478050b8e-data-volume\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.000570 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb1cb847-7b39-496c-ab68-ed6478050b8e-crio-socket\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.000707 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a283d73-3254-49ee-bd3d-32015aea25e0-config-volume\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.000707 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb1cb847-7b39-496c-ab68-ed6478050b8e-data-volume\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.000806 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000758    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a283d73-3254-49ee-bd3d-32015aea25e0-tmp-dir\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.000961 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.000938    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb1cb847-7b39-496c-ab68-ed6478050b8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.001097 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.001080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8a283d73-3254-49ee-bd3d-32015aea25e0-config-volume\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.004334 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.004314    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a283d73-3254-49ee-bd3d-32015aea25e0-metrics-tls\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.004334 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.004324    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb1cb847-7b39-496c-ab68-ed6478050b8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.009130 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.009104    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwsq2\" (UniqueName: \"kubernetes.io/projected/eb1cb847-7b39-496c-ab68-ed6478050b8e-kube-api-access-rwsq2\") pod \"insights-runtime-extractor-njphq\" (UID: \"eb1cb847-7b39-496c-ab68-ed6478050b8e\") " pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.009130 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.009124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmj8z\" (UniqueName: \"kubernetes.io/projected/8a283d73-3254-49ee-bd3d-32015aea25e0-kube-api-access-xmj8z\") pod \"dns-default-x6fbg\" (UID: \"8a283d73-3254-49ee-bd3d-32015aea25e0\") " pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.088920 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.088882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-njphq"
Jul 22 02:29:12.101206 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.101173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f191e60-ef7e-4fc1-93da-9081634c8ba4-cert\") pod \"ingress-canary-vx8b4\" (UID: \"1f191e60-ef7e-4fc1-93da-9081634c8ba4\") " pod="openshift-ingress-canary/ingress-canary-vx8b4"
Jul 22 02:29:12.101343 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.101230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclb8\" (UniqueName: \"kubernetes.io/projected/1f191e60-ef7e-4fc1-93da-9081634c8ba4-kube-api-access-sclb8\") pod \"ingress-canary-vx8b4\" (UID: \"1f191e60-ef7e-4fc1-93da-9081634c8ba4\") " pod="openshift-ingress-canary/ingress-canary-vx8b4"
Jul 22 02:29:12.103681 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.103650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f191e60-ef7e-4fc1-93da-9081634c8ba4-cert\") pod \"ingress-canary-vx8b4\" (UID: \"1f191e60-ef7e-4fc1-93da-9081634c8ba4\") " pod="openshift-ingress-canary/ingress-canary-vx8b4"
Jul 22 02:29:12.110367 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.110347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclb8\" (UniqueName: \"kubernetes.io/projected/1f191e60-ef7e-4fc1-93da-9081634c8ba4-kube-api-access-sclb8\") pod \"ingress-canary-vx8b4\" (UID: \"1f191e60-ef7e-4fc1-93da-9081634c8ba4\") " pod="openshift-ingress-canary/ingress-canary-vx8b4"
Jul 22 02:29:12.112192 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.112176    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:12.135364 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.135338    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vx8b4"
Jul 22 02:29:12.305094 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.305065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x6fbg"]
Jul 22 02:29:12.308844 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:12.308817    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a283d73_3254_49ee_bd3d_32015aea25e0.slice/crio-ac44f6621158d0319cd592946ef1249d52a730a48daefba713fc515d3bfa6df9 WatchSource:0}: Error finding container ac44f6621158d0319cd592946ef1249d52a730a48daefba713fc515d3bfa6df9: Status 404 returned error can't find the container with id ac44f6621158d0319cd592946ef1249d52a730a48daefba713fc515d3bfa6df9
Jul 22 02:29:12.315329 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.315305    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-njphq"]
Jul 22 02:29:12.319836 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:12.319813    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1cb847_7b39_496c_ab68_ed6478050b8e.slice/crio-3e20ea3d7ff4440c82dae16b3ea7bd7b6a94409932024cf91a18a6aa6ee083b5 WatchSource:0}: Error finding container 3e20ea3d7ff4440c82dae16b3ea7bd7b6a94409932024cf91a18a6aa6ee083b5: Status 404 returned error can't find the container with id 3e20ea3d7ff4440c82dae16b3ea7bd7b6a94409932024cf91a18a6aa6ee083b5
Jul 22 02:29:12.333893 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.333865    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vx8b4"]
Jul 22 02:29:12.337798 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:12.337775    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f191e60_ef7e_4fc1_93da_9081634c8ba4.slice/crio-381b926383e717c26e79ac0bb064f460f66b87b37877c5653d4d308f0501d673 WatchSource:0}: Error finding container 381b926383e717c26e79ac0bb064f460f66b87b37877c5653d4d308f0501d673: Status 404 returned error can't find the container with id 381b926383e717c26e79ac0bb064f460f66b87b37877c5653d4d308f0501d673
Jul 22 02:29:12.867277 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.867246    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:12.867507 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.867361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:12.867507 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.867393    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:12.870255 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.870232    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 02:29:12.870732 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.870714    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-slk65\""
Jul 22 02:29:12.871311 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.871291    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kb24x\""
Jul 22 02:29:12.871515 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.871499    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 02:29:12.872047 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.872010    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 02:29:12.872138 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:12.872046    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 02:29:13.068359 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:13.068285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6fbg" event={"ID":"8a283d73-3254-49ee-bd3d-32015aea25e0","Type":"ContainerStarted","Data":"ac44f6621158d0319cd592946ef1249d52a730a48daefba713fc515d3bfa6df9"}
Jul 22 02:29:13.070686 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:13.070651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njphq" event={"ID":"eb1cb847-7b39-496c-ab68-ed6478050b8e","Type":"ContainerStarted","Data":"3e20ea3d7ff4440c82dae16b3ea7bd7b6a94409932024cf91a18a6aa6ee083b5"}
Jul 22 02:29:13.072907 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:13.072878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vx8b4" event={"ID":"1f191e60-ef7e-4fc1-93da-9081634c8ba4","Type":"ContainerStarted","Data":"381b926383e717c26e79ac0bb064f460f66b87b37877c5653d4d308f0501d673"}
Jul 22 02:29:20.093980 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:20.093936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vx8b4" event={"ID":"1f191e60-ef7e-4fc1-93da-9081634c8ba4","Type":"ContainerStarted","Data":"341df211b103a63ddbdf4f62181b6760c182943437e0619b97eb480c65664b00"}
Jul 22 02:29:20.096581 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:20.096548    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6fbg" event={"ID":"8a283d73-3254-49ee-bd3d-32015aea25e0","Type":"ContainerStarted","Data":"f177f8f543bae38be8a5a67ab54e0845d3dd15fe90c018c90f616a617fd6e3a2"}
Jul 22 02:29:20.096720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:20.096601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x6fbg" event={"ID":"8a283d73-3254-49ee-bd3d-32015aea25e0","Type":"ContainerStarted","Data":"84256c1e9bbad54e368b9f26ce39445caddd4a9e1e60478a1464b61dfe6a7a20"}
Jul 22 02:29:20.096720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:20.096627    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:20.098312 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:20.098283    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njphq" event={"ID":"eb1cb847-7b39-496c-ab68-ed6478050b8e","Type":"ContainerStarted","Data":"29ef9f500468d5de9c38eb72b2cd5f1d2bca0822a2a47863f0823f703401b2c4"}
Jul 22 02:29:20.112778 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:20.112727    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vx8b4" podStartSLOduration=2.34922101 podStartE2EDuration="9.112709287s" totalImagesPullingTime="6.763488277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:20.111628564 +0000 UTC m=+49.792361611" watchObservedRunningTime="2026-07-22 02:29:20.112709287 +0000 UTC m=+49.793442351"
Jul 22 02:29:20.136363 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:20.136300    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x6fbg" podStartSLOduration=2.344112916 podStartE2EDuration="9.136279447s" totalImagesPullingTime="6.792166531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:20.13573932 +0000 UTC m=+49.816472370" watchObservedRunningTime="2026-07-22 02:29:20.136279447 +0000 UTC m=+49.817012494"
Jul 22 02:29:22.113176 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.113144    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"]
Jul 22 02:29:22.267767 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.267735    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"]
Jul 22 02:29:22.267913 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.267895    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.271604 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.271571    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 22 02:29:22.271759 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.271609    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 22 02:29:22.271759 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.271618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-465bm\""
Jul 22 02:29:22.271759 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.271571    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 22 02:29:22.271759 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.271574    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 22 02:29:22.271759 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.271574    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 22 02:29:22.277897 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.277870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.278011 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.277911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.278011 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.277935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ecbb10-336d-4848-ad50-f7e35431b349-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.278113 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.278019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srm2k\" (UniqueName: \"kubernetes.io/projected/29ecbb10-336d-4848-ad50-f7e35431b349-kube-api-access-srm2k\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.378903 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.378814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.378903 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.378849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.378903 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.378867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ecbb10-336d-4848-ad50-f7e35431b349-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.378903 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.378907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srm2k\" (UniqueName: \"kubernetes.io/projected/29ecbb10-336d-4848-ad50-f7e35431b349-kube-api-access-srm2k\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.379261 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:22.378961    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 22 02:29:22.379261 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:22.379101    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-tls podName:29ecbb10-336d-4848-ad50-f7e35431b349 nodeName:}" failed. No retries permitted until 2026-07-22 02:29:22.879015856 +0000 UTC m=+52.559748884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-8jg7g" (UID: "29ecbb10-336d-4848-ad50-f7e35431b349") : secret "prometheus-operator-tls" not found
Jul 22 02:29:22.379795 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.379773    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29ecbb10-336d-4848-ad50-f7e35431b349-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.381290 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.381269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.388262 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.388238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srm2k\" (UniqueName: \"kubernetes.io/projected/29ecbb10-336d-4848-ad50-f7e35431b349-kube-api-access-srm2k\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.882512 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.882475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:22.885236 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:22.885209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/29ecbb10-336d-4848-ad50-f7e35431b349-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8jg7g\" (UID: \"29ecbb10-336d-4848-ad50-f7e35431b349\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:23.178463 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:23.178364    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"
Jul 22 02:29:23.630961 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:23.630932    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g"]
Jul 22 02:29:23.801320 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:23.801228    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ecbb10_336d_4848_ad50_f7e35431b349.slice/crio-5aa3252d85d73e274e5a928f9f9761e47553dd933cd49578d8c796dd2a8405cd WatchSource:0}: Error finding container 5aa3252d85d73e274e5a928f9f9761e47553dd933cd49578d8c796dd2a8405cd: Status 404 returned error can't find the container with id 5aa3252d85d73e274e5a928f9f9761e47553dd933cd49578d8c796dd2a8405cd
Jul 22 02:29:24.111252 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:24.111158    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njphq" event={"ID":"eb1cb847-7b39-496c-ab68-ed6478050b8e","Type":"ContainerStarted","Data":"a0e52bcdec3e5c62239f2a98f5177db611562aa9bb82e3fc3542a86fda2a4cd0"}
Jul 22 02:29:24.112207 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:24.112187    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g" event={"ID":"29ecbb10-336d-4848-ad50-f7e35431b349","Type":"ContainerStarted","Data":"5aa3252d85d73e274e5a928f9f9761e47553dd933cd49578d8c796dd2a8405cd"}
Jul 22 02:29:26.120620 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:26.120583    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g" event={"ID":"29ecbb10-336d-4848-ad50-f7e35431b349","Type":"ContainerStarted","Data":"c7d0326d663c9518037037d5685b00cdc3926b34bcb97d20789d7464ba0a585a"}
Jul 22 02:29:26.121082 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:26.120648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g" event={"ID":"29ecbb10-336d-4848-ad50-f7e35431b349","Type":"ContainerStarted","Data":"798d276843b9a475391ac4bfb4bfba66273fae46103fc18dc45a9f79706a0e36"}
Jul 22 02:29:26.140890 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:26.140833    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8jg7g" podStartSLOduration=2.664564392 podStartE2EDuration="4.14081078s" totalImagesPullingTime="1.476246388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:26.139959596 +0000 UTC m=+55.820692643" watchObservedRunningTime="2026-07-22 02:29:26.14081078 +0000 UTC m=+55.821543820"
Jul 22 02:29:28.130381 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.130341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-njphq" event={"ID":"eb1cb847-7b39-496c-ab68-ed6478050b8e","Type":"ContainerStarted","Data":"acbb1953b710c853e39441eff34f7b36875c8408b6b23230947c998df2140e97"}
Jul 22 02:29:28.158063 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.157749    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-njphq" podStartSLOduration=2.246509288 podStartE2EDuration="17.157728366s" totalImagesPullingTime="14.911219078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:29:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:28.157281528 +0000 UTC m=+57.838014574" watchObservedRunningTime="2026-07-22 02:29:28.157728366 +0000 UTC m=+57.838461413"
Jul 22 02:29:28.519493 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.519460    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c62tt"]
Jul 22 02:29:28.524950 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.524925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.527054 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.527019    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 22 02:29:28.527350 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.527332    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 22 02:29:28.527464 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.527334    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 22 02:29:28.527633 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.527612    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z2bb6\""
Jul 22 02:29:28.626521 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626479    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qgj8\" (UniqueName: \"kubernetes.io/projected/8c5ac591-dd52-4dc6-b246-b71761defe9f-kube-api-access-8qgj8\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.626521 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-textfile\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.626761 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-tls\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.626761 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626590    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.626761 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-root\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.626761 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-wtmp\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.626761 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626687    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c5ac591-dd52-4dc6-b246-b71761defe9f-metrics-client-ca\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.626943 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.626943 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.626808    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-sys\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.727589 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.727550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-tls\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.727745 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.727604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.727745 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:28.727724    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 22 02:29:28.727824 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:28.727795    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-tls podName:8c5ac591-dd52-4dc6-b246-b71761defe9f nodeName:}" failed. No retries permitted until 2026-07-22 02:29:29.22777609 +0000 UTC m=+58.908509115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-tls") pod "node-exporter-c62tt" (UID: "8c5ac591-dd52-4dc6-b246-b71761defe9f") : secret "node-exporter-tls" not found
Jul 22 02:29:28.727883 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.727827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-root\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.727883 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.727860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-wtmp\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.727883 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.727879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c5ac591-dd52-4dc6-b246-b71761defe9f-metrics-client-ca\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728010 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.727932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728010 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.727960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-root\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728010 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.727977    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-sys\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728010 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.728004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qgj8\" (UniqueName: \"kubernetes.io/projected/8c5ac591-dd52-4dc6-b246-b71761defe9f-kube-api-access-8qgj8\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728231 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.728022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-textfile\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728231 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.728085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-sys\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728231 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.728107    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-wtmp\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728231 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.728177    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728574 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.728378    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-textfile\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.728574 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.728454    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c5ac591-dd52-4dc6-b246-b71761defe9f-metrics-client-ca\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.730627 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.730606    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:28.738481 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:28.738453    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qgj8\" (UniqueName: \"kubernetes.io/projected/8c5ac591-dd52-4dc6-b246-b71761defe9f-kube-api-access-8qgj8\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:29.231985 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.231949    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-tls\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:29.234408 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.234388    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c5ac591-dd52-4dc6-b246-b71761defe9f-node-exporter-tls\") pod \"node-exporter-c62tt\" (UID: \"8c5ac591-dd52-4dc6-b246-b71761defe9f\") " pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:29.433651 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.433613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c62tt"
Jul 22 02:29:29.444349 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:29.444322    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5ac591_dd52_4dc6_b246_b71761defe9f.slice/crio-e705337e126364a67b4687e088af7d3e60c479aeb0fb862b1c6d81f993f400f9 WatchSource:0}: Error finding container e705337e126364a67b4687e088af7d3e60c479aeb0fb862b1c6d81f993f400f9: Status 404 returned error can't find the container with id e705337e126364a67b4687e088af7d3e60c479aeb0fb862b1c6d81f993f400f9
Jul 22 02:29:29.616911 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.616821    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 22 02:29:29.619820 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.619791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.622301 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.622281    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 22 02:29:29.623457 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.623438    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 22 02:29:29.623660 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.623646    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 22 02:29:29.623737 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.623666    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 22 02:29:29.623879 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.623865    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 22 02:29:29.623879 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.623876    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 22 02:29:29.623997 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.623909    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 22 02:29:29.623997 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.623968    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gqwng\""
Jul 22 02:29:29.624272 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.624258    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 22 02:29:29.624472 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.624458    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 22 02:29:29.635216 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4af2adf2-6cfd-458a-b722-e9fc30649923-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635322 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4af2adf2-6cfd-458a-b722-e9fc30649923-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635322 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635322 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635275    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69xfn\" (UniqueName: \"kubernetes.io/projected/4af2adf2-6cfd-458a-b722-e9fc30649923-kube-api-access-69xfn\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635322 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635295    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-config-volume\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635511 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635511 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af2adf2-6cfd-458a-b722-e9fc30649923-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635511 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-web-config\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635511 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635704 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635704 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4af2adf2-6cfd-458a-b722-e9fc30649923-config-out\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635704 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.635704 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.635632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4af2adf2-6cfd-458a-b722-e9fc30649923-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.638352 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.638333    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 22 02:29:29.736236 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af2adf2-6cfd-458a-b722-e9fc30649923-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-web-config\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736550 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736550 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4af2adf2-6cfd-458a-b722-e9fc30649923-config-out\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736550 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736486    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736550 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4af2adf2-6cfd-458a-b722-e9fc30649923-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736738 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4af2adf2-6cfd-458a-b722-e9fc30649923-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736738 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4af2adf2-6cfd-458a-b722-e9fc30649923-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736738 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736640    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736738 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736674    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xfn\" (UniqueName: \"kubernetes.io/projected/4af2adf2-6cfd-458a-b722-e9fc30649923-kube-api-access-69xfn\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736738 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-config-volume\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736981 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.736981 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.736908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4af2adf2-6cfd-458a-b722-e9fc30649923-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.737159 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:29.737008    2583 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 22 02:29:29.737285 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:29.737239    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-main-tls podName:4af2adf2-6cfd-458a-b722-e9fc30649923 nodeName:}" failed. No retries permitted until 2026-07-22 02:29:30.237096089 +0000 UTC m=+59.917829121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "4af2adf2-6cfd-458a-b722-e9fc30649923") : secret "alertmanager-main-tls" not found
Jul 22 02:29:29.737590 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.737482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4af2adf2-6cfd-458a-b722-e9fc30649923-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.738817 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.738773    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4af2adf2-6cfd-458a-b722-e9fc30649923-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.739724 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.739694    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.739826 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.739703    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-web-config\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.740117 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.740064    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.740253 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.740230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4af2adf2-6cfd-458a-b722-e9fc30649923-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.740366 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.740314    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.740530 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.740514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-config-volume\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.740657 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.740635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.741069 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.741015    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4af2adf2-6cfd-458a-b722-e9fc30649923-config-out\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:29.745291 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:29.745257    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xfn\" (UniqueName: \"kubernetes.io/projected/4af2adf2-6cfd-458a-b722-e9fc30649923-kube-api-access-69xfn\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:30.040619 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:30.040592    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xxrcb"
Jul 22 02:29:30.103944 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:30.103914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x6fbg"
Jul 22 02:29:30.143596 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:30.143545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c62tt" event={"ID":"8c5ac591-dd52-4dc6-b246-b71761defe9f","Type":"ContainerStarted","Data":"e705337e126364a67b4687e088af7d3e60c479aeb0fb862b1c6d81f993f400f9"}
Jul 22 02:29:30.243400 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:30.243365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:30.246188 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:30.246163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4af2adf2-6cfd-458a-b722-e9fc30649923-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4af2adf2-6cfd-458a-b722-e9fc30649923\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:30.531222 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:30.531179    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 22 02:29:30.679403 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:30.679331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 22 02:29:30.682455 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:30.682430    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af2adf2_6cfd_458a_b722_e9fc30649923.slice/crio-fad4ec1f75f57ff1c8e02866954032ab973c198249f062f8203e18266abcaa84 WatchSource:0}: Error finding container fad4ec1f75f57ff1c8e02866954032ab973c198249f062f8203e18266abcaa84: Status 404 returned error can't find the container with id fad4ec1f75f57ff1c8e02866954032ab973c198249f062f8203e18266abcaa84
Jul 22 02:29:31.147850 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:31.147815    2583 generic.go:356] "Generic (PLEG): container finished" podID="8c5ac591-dd52-4dc6-b246-b71761defe9f" containerID="a81e9e4c65d86066845085b3ccd9bf4718ad34e28a01fb2156b4ba91bf88b4c7" exitCode=0
Jul 22 02:29:31.148059 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:31.147908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c62tt" event={"ID":"8c5ac591-dd52-4dc6-b246-b71761defe9f","Type":"ContainerDied","Data":"a81e9e4c65d86066845085b3ccd9bf4718ad34e28a01fb2156b4ba91bf88b4c7"}
Jul 22 02:29:31.149188 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:31.149165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4af2adf2-6cfd-458a-b722-e9fc30649923","Type":"ContainerStarted","Data":"fad4ec1f75f57ff1c8e02866954032ab973c198249f062f8203e18266abcaa84"}
Jul 22 02:29:32.154073 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:32.153966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c62tt" event={"ID":"8c5ac591-dd52-4dc6-b246-b71761defe9f","Type":"ContainerStarted","Data":"30e08cb1a044af18c5076bf9895f575cf743b0e322c8138cb82f9de1c8d6e9ff"}
Jul 22 02:29:32.154476 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:32.154076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c62tt" event={"ID":"8c5ac591-dd52-4dc6-b246-b71761defe9f","Type":"ContainerStarted","Data":"031ce55892d15728919bd99e5d60f82fff7cb97626caf10095b81e2608e5a815"}
Jul 22 02:29:32.155516 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:32.155490    2583 generic.go:356] "Generic (PLEG): container finished" podID="4af2adf2-6cfd-458a-b722-e9fc30649923" containerID="9f04799d0dd504cd77ba0a3dc67ce2409631727f3108ac364291665d25f74989" exitCode=0
Jul 22 02:29:32.155622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:32.155530    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4af2adf2-6cfd-458a-b722-e9fc30649923","Type":"ContainerDied","Data":"9f04799d0dd504cd77ba0a3dc67ce2409631727f3108ac364291665d25f74989"}
Jul 22 02:29:32.174126 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:32.174075    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c62tt" podStartSLOduration=3.119829818 podStartE2EDuration="4.174060241s" totalImagesPullingTime="980.698262ms" totalInitContainerRuntime="73.532161ms" isStatefulPod=true podCreationTimestamp="2026-07-22 02:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:32.173538555 +0000 UTC m=+61.854271592" watchObservedRunningTime="2026-07-22 02:29:32.174060241 +0000 UTC m=+61.854793283"
Jul 22 02:29:33.271355 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.271322    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"]
Jul 22 02:29:33.273965 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.273938    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:33.276705 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.276680    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gk8s4\""
Jul 22 02:29:33.276834 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.276693    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 22 02:29:33.285661 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.285634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"]
Jul 22 02:29:33.368916 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.368883    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-mhm8p\" (UID: \"d17ff1de-43a2-4436-bc0f-3a2cde1dd356\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:33.369155 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.368957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qz4z\" (UniqueName: \"kubernetes.io/projected/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-kube-api-access-6qz4z\") pod \"monitoring-plugin-7b7b7bc8bf-mhm8p\" (UID: \"d17ff1de-43a2-4436-bc0f-3a2cde1dd356\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:33.402353 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.402318    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fc9c8994f-phkhc"]
Jul 22 02:29:33.405182 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.405163    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.409082 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.408992    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 22 02:29:33.409082 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.409053    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 22 02:29:33.409441 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.409089    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 22 02:29:33.409441 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.409283    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 22 02:29:33.409441 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.409292    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q7fvk\""
Jul 22 02:29:33.409441 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.409349    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 22 02:29:33.409441 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.409381    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 22 02:29:33.409693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.409673    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 22 02:29:33.415361 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.415343    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 22 02:29:33.415995 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.415973    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fc9c8994f-phkhc"]
Jul 22 02:29:33.470416 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-oauth-config\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.470609 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qz4z\" (UniqueName: \"kubernetes.io/projected/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-kube-api-access-6qz4z\") pod \"monitoring-plugin-7b7b7bc8bf-mhm8p\" (UID: \"d17ff1de-43a2-4436-bc0f-3a2cde1dd356\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:33.470609 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470471    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-serving-cert\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.470609 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-config\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.470609 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzts\" (UniqueName: \"kubernetes.io/projected/775f34e9-e595-43c9-bda2-3a5e1fd71813-kube-api-access-6lzts\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.470609 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-service-ca\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.470837 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-oauth-serving-cert\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.470837 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-mhm8p\" (UID: \"d17ff1de-43a2-4436-bc0f-3a2cde1dd356\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:33.470837 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.470766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-trusted-ca-bundle\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.470958 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:33.470883    2583 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 22 02:29:33.470958 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:29:33.470952    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-monitoring-plugin-cert podName:d17ff1de-43a2-4436-bc0f-3a2cde1dd356 nodeName:}" failed. No retries permitted until 2026-07-22 02:29:33.970931494 +0000 UTC m=+63.651664517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-mhm8p" (UID: "d17ff1de-43a2-4436-bc0f-3a2cde1dd356") : secret "monitoring-plugin-cert" not found
Jul 22 02:29:33.482234 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.482204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qz4z\" (UniqueName: \"kubernetes.io/projected/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-kube-api-access-6qz4z\") pod \"monitoring-plugin-7b7b7bc8bf-mhm8p\" (UID: \"d17ff1de-43a2-4436-bc0f-3a2cde1dd356\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:33.571325 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.571250    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-config\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.571325 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.571293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzts\" (UniqueName: \"kubernetes.io/projected/775f34e9-e595-43c9-bda2-3a5e1fd71813-kube-api-access-6lzts\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.571325 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.571314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-service-ca\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.571595 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.571332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-oauth-serving-cert\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.571595 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.571385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-trusted-ca-bundle\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.571595 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.571480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-oauth-config\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.571595 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.571571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-serving-cert\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.572070 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.572018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-config\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.572217 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.572199    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-oauth-serving-cert\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.572293 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.572253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-service-ca\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.572376 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.572351    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-trusted-ca-bundle\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.574807 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.574787    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-serving-cert\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.575106 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.575089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-oauth-config\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.580124 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.580101    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzts\" (UniqueName: \"kubernetes.io/projected/775f34e9-e595-43c9-bda2-3a5e1fd71813-kube-api-access-6lzts\") pod \"console-6fc9c8994f-phkhc\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") " pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.717151 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.717124    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:33.868010 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.867968    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fc9c8994f-phkhc"]
Jul 22 02:29:33.871852 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:33.871832    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775f34e9_e595_43c9_bda2_3a5e1fd71813.slice/crio-b67fa4e6f2960a8434954da43d2cd167f343e06260f72d673d26b6ac90e66dd6 WatchSource:0}: Error finding container b67fa4e6f2960a8434954da43d2cd167f343e06260f72d673d26b6ac90e66dd6: Status 404 returned error can't find the container with id b67fa4e6f2960a8434954da43d2cd167f343e06260f72d673d26b6ac90e66dd6
Jul 22 02:29:33.974381 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.974349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-mhm8p\" (UID: \"d17ff1de-43a2-4436-bc0f-3a2cde1dd356\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:33.976794 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:33.976771    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d17ff1de-43a2-4436-bc0f-3a2cde1dd356-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-mhm8p\" (UID: \"d17ff1de-43a2-4436-bc0f-3a2cde1dd356\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:34.166361 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.166277    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4af2adf2-6cfd-458a-b722-e9fc30649923","Type":"ContainerStarted","Data":"351e2040f83c24766a1fdb9987508f056dc81e5cf13d51a238754ed9bc5f10ce"}
Jul 22 02:29:34.166361 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.166341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4af2adf2-6cfd-458a-b722-e9fc30649923","Type":"ContainerStarted","Data":"2557e78ee2e78b940a9bbc4636714e7140ea81dece6f56ec30c46f9aad3552fe"}
Jul 22 02:29:34.166566 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.166369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4af2adf2-6cfd-458a-b722-e9fc30649923","Type":"ContainerStarted","Data":"32addf1dd1cfdfbce31edfbe4450d15e7c4c9ce83e5b4ff3c2b8395d27ffb926"}
Jul 22 02:29:34.166566 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.166406    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4af2adf2-6cfd-458a-b722-e9fc30649923","Type":"ContainerStarted","Data":"fd74a10221a74c033637aefb245f937ad548fb318152b5162ca37b2c970c7015"}
Jul 22 02:29:34.166566 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.166433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4af2adf2-6cfd-458a-b722-e9fc30649923","Type":"ContainerStarted","Data":"57b2abb2d4017b6a8174738f3f704c1eaa0c4f1f1c8f9050888ad3bc57b86887"}
Jul 22 02:29:34.167556 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.167531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fc9c8994f-phkhc" event={"ID":"775f34e9-e595-43c9-bda2-3a5e1fd71813","Type":"ContainerStarted","Data":"b67fa4e6f2960a8434954da43d2cd167f343e06260f72d673d26b6ac90e66dd6"}
Jul 22 02:29:34.185005 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.184981    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:34.309633 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.309603    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"]
Jul 22 02:29:34.313153 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:34.313116    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17ff1de_43a2_4436_bc0f_3a2cde1dd356.slice/crio-ec91771e894ea5181a05f39d855bd8c52583c1a47dd06bdd5c9a66cd79f196a8 WatchSource:0}: Error finding container ec91771e894ea5181a05f39d855bd8c52583c1a47dd06bdd5c9a66cd79f196a8: Status 404 returned error can't find the container with id ec91771e894ea5181a05f39d855bd8c52583c1a47dd06bdd5c9a66cd79f196a8
Jul 22 02:29:34.746479 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.746434    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 22 02:29:34.750570 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.750544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.758057 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.757458    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 22 02:29:34.758057 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.757801    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 22 02:29:34.758057 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.757939    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 22 02:29:34.758057 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.758011    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 22 02:29:34.758440 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.758186    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 22 02:29:34.758440 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.758428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 22 02:29:34.761067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.758635    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 22 02:29:34.761067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.758744    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 22 02:29:34.761067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.758960    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 22 02:29:34.761067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.759376    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 22 02:29:34.761067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.759572    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 22 02:29:34.761067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.759780    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 22 02:29:34.761067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.759983    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 22 02:29:34.761067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.760378    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z4798\""
Jul 22 02:29:34.763059 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.762533    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 22 02:29:34.763059 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.762731    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 22 02:29:34.763059 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.759376    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7ldgjh6569mud\""
Jul 22 02:29:34.771475 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.771444    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 22 02:29:34.784709 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784677    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.784854 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.784854 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-config-out\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.784854 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784784    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-config\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.784854 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785103 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785103 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784887    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785103 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784919    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785103 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785103 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.784979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-web-config\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785103 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785103 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785127    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-955cd\" (UniqueName: \"kubernetes.io/projected/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-kube-api-access-955cd\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.785425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.785389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.886947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.886993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887132    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887238    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-955cd\" (UniqueName: \"kubernetes.io/projected/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-kube-api-access-955cd\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-config-out\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-config\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.887622 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.888562 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887500    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.888562 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.888562 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.887561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-web-config\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.888562 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.888466    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.894552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.889173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.894552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.889479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.894552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.889601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.894552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.893544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.894552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.894259    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.898264 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.897824    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.899308 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.898862    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.903720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.903692    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-web-config\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.903943 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.903923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.907262 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.906060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.907262 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.906068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.907262 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.906295    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.907262 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.906710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.909449 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.907017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-955cd\" (UniqueName: \"kubernetes.io/projected/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-kube-api-access-955cd\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.913650 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.913609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-config\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.914002 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.913978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.915207 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.915154    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.915815 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.915796    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:34.915935 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:34.915888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10-config-out\") pod \"prometheus-k8s-0\" (UID: \"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:35.071418 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:35.071335    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:29:35.175517 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:35.175387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p" event={"ID":"d17ff1de-43a2-4436-bc0f-3a2cde1dd356","Type":"ContainerStarted","Data":"ec91771e894ea5181a05f39d855bd8c52583c1a47dd06bdd5c9a66cd79f196a8"}
Jul 22 02:29:35.301536 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:35.301503    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 22 02:29:35.311845 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:35.311814    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79cfd6e_8b5b_442e_baf2_4eebc5b7ee10.slice/crio-20c246972f675e9a9d659ecf0da9afc44065d03c7f29b13dd503c16e188c9f35 WatchSource:0}: Error finding container 20c246972f675e9a9d659ecf0da9afc44065d03c7f29b13dd503c16e188c9f35: Status 404 returned error can't find the container with id 20c246972f675e9a9d659ecf0da9afc44065d03c7f29b13dd503c16e188c9f35
Jul 22 02:29:36.180796 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.180707    2583 generic.go:356] "Generic (PLEG): container finished" podID="b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10" containerID="775c0be2ca90a114c36b4bcae3f71578b2d7dc3eb375216ef69125429c1e3987" exitCode=0
Jul 22 02:29:36.180953 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.180801    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10","Type":"ContainerDied","Data":"775c0be2ca90a114c36b4bcae3f71578b2d7dc3eb375216ef69125429c1e3987"}
Jul 22 02:29:36.180953 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.180868    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10","Type":"ContainerStarted","Data":"20c246972f675e9a9d659ecf0da9afc44065d03c7f29b13dd503c16e188c9f35"}
Jul 22 02:29:36.184517 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.184493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4af2adf2-6cfd-458a-b722-e9fc30649923","Type":"ContainerStarted","Data":"c6b23325f6e7518d0f8f54e96f60470619a04dc8d6867cb3697a6df0cafcbcee"}
Jul 22 02:29:36.243502 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.243444    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.342687198 podStartE2EDuration="7.243424585s" totalImagesPullingTime="3.884376684s" totalInitContainerRuntime="16.360703ms" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:36.241903207 +0000 UTC m=+65.922636266" watchObservedRunningTime="2026-07-22 02:29:36.243424585 +0000 UTC m=+65.924157609"
Jul 22 02:29:36.605054 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.605000    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:36.605509 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.605093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:36.605509 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.605146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:36.607673 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.607647    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 02:29:36.607673 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.607674    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 02:29:36.607833 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.607750    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 02:29:36.618316 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.618288    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 02:29:36.618316 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.618307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cdcb05cc-be7e-40e0-943d-e84bc42a16dc-original-pull-secret\") pod \"global-pull-secret-syncer-6grr5\" (UID: \"cdcb05cc-be7e-40e0-943d-e84bc42a16dc\") " pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:36.618476 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.618447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b9f796-93fd-4c18-a461-4894a9ce9d0c-metrics-certs\") pod \"network-metrics-daemon-5t4kv\" (UID: \"b1b9f796-93fd-4c18-a461-4894a9ce9d0c\") " pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:36.629705 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.629675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhlsq\" (UniqueName: \"kubernetes.io/projected/ebfaa30b-0d15-46e0-91ed-b59c86704f8f-kube-api-access-bhlsq\") pod \"network-check-target-4gnks\" (UID: \"ebfaa30b-0d15-46e0-91ed-b59c86704f8f\") " pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:36.880972 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.880896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6grr5"
Jul 22 02:29:36.892614 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.892580    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kb24x\""
Jul 22 02:29:36.898848 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.898815    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-slk65\""
Jul 22 02:29:36.900680 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.900655    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:36.906967 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:36.906948    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5t4kv"
Jul 22 02:29:37.549411 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:37.549316    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5t4kv"]
Jul 22 02:29:37.554722 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:37.554661    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b9f796_93fd_4c18_a461_4894a9ce9d0c.slice/crio-aad042222ca14e059c4d326ed8b2877f0469bff5fb652eea86f057a15c7f3ddd WatchSource:0}: Error finding container aad042222ca14e059c4d326ed8b2877f0469bff5fb652eea86f057a15c7f3ddd: Status 404 returned error can't find the container with id aad042222ca14e059c4d326ed8b2877f0469bff5fb652eea86f057a15c7f3ddd
Jul 22 02:29:37.577701 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:37.577626    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6grr5"]
Jul 22 02:29:37.578792 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:37.578764    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gnks"]
Jul 22 02:29:37.581976 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:37.581943    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcb05cc_be7e_40e0_943d_e84bc42a16dc.slice/crio-0d35f56919d06aa7a4d85b6ae913c414e80388de6785ecc6d2b8a1ad28899a69 WatchSource:0}: Error finding container 0d35f56919d06aa7a4d85b6ae913c414e80388de6785ecc6d2b8a1ad28899a69: Status 404 returned error can't find the container with id 0d35f56919d06aa7a4d85b6ae913c414e80388de6785ecc6d2b8a1ad28899a69
Jul 22 02:29:37.582300 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:29:37.582271    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfaa30b_0d15_46e0_91ed_b59c86704f8f.slice/crio-571b061fcd0e3f59004d39da6623c97b10b939ddc7c8ea9d4cf2b6f939f8e55b WatchSource:0}: Error finding container 571b061fcd0e3f59004d39da6623c97b10b939ddc7c8ea9d4cf2b6f939f8e55b: Status 404 returned error can't find the container with id 571b061fcd0e3f59004d39da6623c97b10b939ddc7c8ea9d4cf2b6f939f8e55b
Jul 22 02:29:38.196350 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.196273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p" event={"ID":"d17ff1de-43a2-4436-bc0f-3a2cde1dd356","Type":"ContainerStarted","Data":"69b471e8dd4a26b765199086cf2ea95463f05835f59f4b7b99fae7ba8834d110"}
Jul 22 02:29:38.197021 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.196721    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:38.199007 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.198346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5t4kv" event={"ID":"b1b9f796-93fd-4c18-a461-4894a9ce9d0c","Type":"ContainerStarted","Data":"aad042222ca14e059c4d326ed8b2877f0469bff5fb652eea86f057a15c7f3ddd"}
Jul 22 02:29:38.201125 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.201095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fc9c8994f-phkhc" event={"ID":"775f34e9-e595-43c9-bda2-3a5e1fd71813","Type":"ContainerStarted","Data":"a64fadb563aa926dfd633e7098826af58e07b6fae4acac9fd6910114822bd404"}
Jul 22 02:29:38.203465 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.203421    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p"
Jul 22 02:29:38.204372 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.204318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6grr5" event={"ID":"cdcb05cc-be7e-40e0-943d-e84bc42a16dc","Type":"ContainerStarted","Data":"0d35f56919d06aa7a4d85b6ae913c414e80388de6785ecc6d2b8a1ad28899a69"}
Jul 22 02:29:38.205896 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.205836    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gnks" event={"ID":"ebfaa30b-0d15-46e0-91ed-b59c86704f8f","Type":"ContainerStarted","Data":"571b061fcd0e3f59004d39da6623c97b10b939ddc7c8ea9d4cf2b6f939f8e55b"}
Jul 22 02:29:38.234973 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.234910    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-mhm8p" podStartSLOduration=2.171487038 podStartE2EDuration="5.234886287s" totalImagesPullingTime="3.063399249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:38.216192295 +0000 UTC m=+67.896925343" watchObservedRunningTime="2026-07-22 02:29:38.234886287 +0000 UTC m=+67.915619334"
Jul 22 02:29:38.253169 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:38.252607    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fc9c8994f-phkhc" podStartSLOduration=1.741577683 podStartE2EDuration="5.252581629s" totalImagesPullingTime="3.511003946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:38.25123702 +0000 UTC m=+67.931970071" watchObservedRunningTime="2026-07-22 02:29:38.252581629 +0000 UTC m=+67.933314676"
Jul 22 02:29:39.218183 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:39.218108    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5t4kv" event={"ID":"b1b9f796-93fd-4c18-a461-4894a9ce9d0c","Type":"ContainerStarted","Data":"ec022a1bbb47c17c8f0da0a2a0a4fb8540232e3b2525270be391993db73a9187"}
Jul 22 02:29:39.218183 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:39.218154    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5t4kv" event={"ID":"b1b9f796-93fd-4c18-a461-4894a9ce9d0c","Type":"ContainerStarted","Data":"7d3bc0a5be420ddd7f39beb137cd3ce4d2ec8dbbaa10ec79d5afd559722c1546"}
Jul 22 02:29:39.243855 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:39.242666    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5t4kv" podStartSLOduration=68.046416685 podStartE2EDuration="1m9.242642128s" totalImagesPullingTime="1.196225443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:39.242001934 +0000 UTC m=+68.922734992" watchObservedRunningTime="2026-07-22 02:29:39.242642128 +0000 UTC m=+68.923375175"
Jul 22 02:29:42.528846 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:42.528809    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fc9c8994f-phkhc"]
Jul 22 02:29:43.717637 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:43.717531    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:29:44.241939 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:44.241901    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10","Type":"ContainerStarted","Data":"d444b2fff1be47be55d29e11966247e76cdf8670f37174d219f30dedf3ff77e1"}
Jul 22 02:29:44.242129 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:44.241991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10","Type":"ContainerStarted","Data":"404850478b404f47df6e5b87647be3ce2dfd3773ac9e67f6801bd17f12ddd908"}
Jul 22 02:29:44.243614 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:44.243574    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6grr5" event={"ID":"cdcb05cc-be7e-40e0-943d-e84bc42a16dc","Type":"ContainerStarted","Data":"a43c32c5d7400d662e9304eb973d77b62c84e8f272bdb2c144abbffc2c383990"}
Jul 22 02:29:44.245395 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:44.245367    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gnks" event={"ID":"ebfaa30b-0d15-46e0-91ed-b59c86704f8f","Type":"ContainerStarted","Data":"13934de0f47cfa6c2238f05c0733d3c8416b771628e8cab58a3ff9b8c7a3f772"}
Jul 22 02:29:44.245521 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:44.245441    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:29:44.260553 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:44.260505    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6grr5" podStartSLOduration=67.380666293 podStartE2EDuration="1m13.260489609s" totalImagesPullingTime="5.879823316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 02:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:44.259177926 +0000 UTC m=+73.939910973" watchObservedRunningTime="2026-07-22 02:29:44.260489609 +0000 UTC m=+73.941222655"
Jul 22 02:29:44.275243 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:44.275205    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4gnks" podStartSLOduration=68.397614163 podStartE2EDuration="1m14.275190335s" totalImagesPullingTime="5.877576172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:44.273889536 +0000 UTC m=+73.954622584" watchObservedRunningTime="2026-07-22 02:29:44.275190335 +0000 UTC m=+73.955923381"
Jul 22 02:29:46.256911 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:46.256874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10","Type":"ContainerStarted","Data":"5024fe7557b1f5832cabb2f13a7bef59e999d4caf64537083083907b9e1fcf11"}
Jul 22 02:29:46.257322 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:46.256940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10","Type":"ContainerStarted","Data":"5b346d060793766742e4e756ff49a4ecdaba6221d03cd1d272da109392c532ab"}
Jul 22 02:29:46.257322 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:46.256965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10","Type":"ContainerStarted","Data":"e0dfba71db49357cbf0be41dd10630106496a9ecba298c423f1d47521282e539"}
Jul 22 02:29:46.257322 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:46.256988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10","Type":"ContainerStarted","Data":"e2958f727cdaf1bb3d2be866554b89d1491d09285c17997943bd08460586d541"}
Jul 22 02:29:46.288148 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:46.288085    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.288673539 podStartE2EDuration="12.288061399s" totalImagesPullingTime="8.973841284s" totalInitContainerRuntime="25.546576ms" isStatefulPod=false podCreationTimestamp="2026-07-22 02:29:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:29:46.285286476 +0000 UTC m=+75.966019514" watchObservedRunningTime="2026-07-22 02:29:46.288061399 +0000 UTC m=+75.968794446"
Jul 22 02:29:50.072336 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:29:50.072291    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:30:07.551443 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:07.551287    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fc9c8994f-phkhc" podUID="775f34e9-e595-43c9-bda2-3a5e1fd71813" containerName="console" containerID="cri-o://a64fadb563aa926dfd633e7098826af58e07b6fae4acac9fd6910114822bd404" gracePeriod=15
Jul 22 02:30:08.341463 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.341436    2583 generic.go:356] "Generic (PLEG): container finished" podID="775f34e9-e595-43c9-bda2-3a5e1fd71813" containerID="a64fadb563aa926dfd633e7098826af58e07b6fae4acac9fd6910114822bd404" exitCode=0
Jul 22 02:30:08.341587 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.341504    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fc9c8994f-phkhc" event={"ID":"775f34e9-e595-43c9-bda2-3a5e1fd71813","Type":"ContainerDied","Data":"a64fadb563aa926dfd633e7098826af58e07b6fae4acac9fd6910114822bd404"}
Jul 22 02:30:08.341587 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.341552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fc9c8994f-phkhc" event={"ID":"775f34e9-e595-43c9-bda2-3a5e1fd71813","Type":"ContainerDied","Data":"b67fa4e6f2960a8434954da43d2cd167f343e06260f72d673d26b6ac90e66dd6"}
Jul 22 02:30:08.341587 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.341567    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b67fa4e6f2960a8434954da43d2cd167f343e06260f72d673d26b6ac90e66dd6"
Jul 22 02:30:08.373852 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.373813    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:30:08.488819 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.488783    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-config\") pod \"775f34e9-e595-43c9-bda2-3a5e1fd71813\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") "
Jul 22 02:30:08.489005 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.488832    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-serving-cert\") pod \"775f34e9-e595-43c9-bda2-3a5e1fd71813\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") "
Jul 22 02:30:08.489005 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.488901    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-trusted-ca-bundle\") pod \"775f34e9-e595-43c9-bda2-3a5e1fd71813\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") "
Jul 22 02:30:08.489005 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.488921    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-oauth-config\") pod \"775f34e9-e595-43c9-bda2-3a5e1fd71813\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") "
Jul 22 02:30:08.489005 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.488935    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-service-ca\") pod \"775f34e9-e595-43c9-bda2-3a5e1fd71813\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") "
Jul 22 02:30:08.489005 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.488959    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lzts\" (UniqueName: \"kubernetes.io/projected/775f34e9-e595-43c9-bda2-3a5e1fd71813-kube-api-access-6lzts\") pod \"775f34e9-e595-43c9-bda2-3a5e1fd71813\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") "
Jul 22 02:30:08.489306 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.489074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-oauth-serving-cert\") pod \"775f34e9-e595-43c9-bda2-3a5e1fd71813\" (UID: \"775f34e9-e595-43c9-bda2-3a5e1fd71813\") "
Jul 22 02:30:08.489306 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.489272    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-config" (OuterVolumeSpecName: "console-config") pod "775f34e9-e595-43c9-bda2-3a5e1fd71813" (UID: "775f34e9-e595-43c9-bda2-3a5e1fd71813"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:30:08.489414 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.489394    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-config\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:30:08.489465 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.489437    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-service-ca" (OuterVolumeSpecName: "service-ca") pod "775f34e9-e595-43c9-bda2-3a5e1fd71813" (UID: "775f34e9-e595-43c9-bda2-3a5e1fd71813"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:30:08.489526 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.489491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "775f34e9-e595-43c9-bda2-3a5e1fd71813" (UID: "775f34e9-e595-43c9-bda2-3a5e1fd71813"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:30:08.489737 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.489652    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "775f34e9-e595-43c9-bda2-3a5e1fd71813" (UID: "775f34e9-e595-43c9-bda2-3a5e1fd71813"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:30:08.491576 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.491540    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "775f34e9-e595-43c9-bda2-3a5e1fd71813" (UID: "775f34e9-e595-43c9-bda2-3a5e1fd71813"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:30:08.491576 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.491549    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775f34e9-e595-43c9-bda2-3a5e1fd71813-kube-api-access-6lzts" (OuterVolumeSpecName: "kube-api-access-6lzts") pod "775f34e9-e595-43c9-bda2-3a5e1fd71813" (UID: "775f34e9-e595-43c9-bda2-3a5e1fd71813"). InnerVolumeSpecName "kube-api-access-6lzts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:30:08.491705 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.491596    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "775f34e9-e595-43c9-bda2-3a5e1fd71813" (UID: "775f34e9-e595-43c9-bda2-3a5e1fd71813"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:30:08.590242 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.590203    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-serving-cert\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:30:08.590242 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.590235    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-trusted-ca-bundle\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:30:08.590242 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.590244    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/775f34e9-e595-43c9-bda2-3a5e1fd71813-console-oauth-config\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:30:08.590242 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.590253    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:30:08.590704 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.590261    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lzts\" (UniqueName: \"kubernetes.io/projected/775f34e9-e595-43c9-bda2-3a5e1fd71813-kube-api-access-6lzts\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:30:08.590704 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:08.590269    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/775f34e9-e595-43c9-bda2-3a5e1fd71813-oauth-serving-cert\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:30:09.345107 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:09.345070    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fc9c8994f-phkhc"
Jul 22 02:30:09.372834 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:09.372800    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fc9c8994f-phkhc"]
Jul 22 02:30:09.383870 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:09.383842    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fc9c8994f-phkhc"]
Jul 22 02:30:10.867782 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:10.867744    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775f34e9-e595-43c9-bda2-3a5e1fd71813" path="/var/lib/kubelet/pods/775f34e9-e595-43c9-bda2-3a5e1fd71813/volumes"
Jul 22 02:30:15.253739 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:15.253698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4gnks"
Jul 22 02:30:35.072384 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:35.072341    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:30:35.093006 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:35.092974    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:30:35.450464 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:35.450394    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 22 02:30:54.289280 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.289244    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77d4f48d9d-fmxv9"]
Jul 22 02:30:54.289672 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.289495    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775f34e9-e595-43c9-bda2-3a5e1fd71813" containerName="console"
Jul 22 02:30:54.289672 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.289506    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="775f34e9-e595-43c9-bda2-3a5e1fd71813" containerName="console"
Jul 22 02:30:54.289672 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.289562    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="775f34e9-e595-43c9-bda2-3a5e1fd71813" containerName="console"
Jul 22 02:30:54.291372 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.291356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.294148 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.294123    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 22 02:30:54.294284 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.294181    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 22 02:30:54.294284 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.294189    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 22 02:30:54.294849 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.294832    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 22 02:30:54.294941 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.294866    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 22 02:30:54.295106 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.295088    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 22 02:30:54.295182 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.295090    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 22 02:30:54.295182 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.295154    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q7fvk\""
Jul 22 02:30:54.298859 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.298584    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 22 02:30:54.304232 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.304213    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d4f48d9d-fmxv9"]
Jul 22 02:30:54.327506 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.327465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-trusted-ca-bundle\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.327506 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.327512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-serving-cert\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.327757 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.327542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-service-ca\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.327757 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.327584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-oauth-serving-cert\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.327757 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.327614    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-oauth-config\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.327757 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.327713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-config\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.327895 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.327760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4sx8\" (UniqueName: \"kubernetes.io/projected/52b8f3e2-445d-4854-8b3d-04b82e1e1473-kube-api-access-z4sx8\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.428553 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.428511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-trusted-ca-bundle\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.428746 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.428564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-serving-cert\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.428746 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.428594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-service-ca\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.428746 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.428615    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-oauth-serving-cert\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.428746 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.428643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-oauth-config\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.428746 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.428679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-config\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.428746 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.428707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4sx8\" (UniqueName: \"kubernetes.io/projected/52b8f3e2-445d-4854-8b3d-04b82e1e1473-kube-api-access-z4sx8\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.429583 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.429442    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-oauth-serving-cert\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.429583 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.429483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-service-ca\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.429583 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.429483    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-config\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.429583 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.429512    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-trusted-ca-bundle\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.431211 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.431189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-oauth-config\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.431354 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.431336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-serving-cert\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.436334 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.436312    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4sx8\" (UniqueName: \"kubernetes.io/projected/52b8f3e2-445d-4854-8b3d-04b82e1e1473-kube-api-access-z4sx8\") pod \"console-77d4f48d9d-fmxv9\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") " pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.601276 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.601185    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:30:54.729427 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:54.729395    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d4f48d9d-fmxv9"]
Jul 22 02:30:54.732918 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:30:54.732890    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b8f3e2_445d_4854_8b3d_04b82e1e1473.slice/crio-2a956e9dcf453ada99b3d606087be9ba285beb59b5a89230896974d34c981080 WatchSource:0}: Error finding container 2a956e9dcf453ada99b3d606087be9ba285beb59b5a89230896974d34c981080: Status 404 returned error can't find the container with id 2a956e9dcf453ada99b3d606087be9ba285beb59b5a89230896974d34c981080
Jul 22 02:30:55.505643 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:55.505610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4f48d9d-fmxv9" event={"ID":"52b8f3e2-445d-4854-8b3d-04b82e1e1473","Type":"ContainerStarted","Data":"eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378"}
Jul 22 02:30:55.506052 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:55.505656    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4f48d9d-fmxv9" event={"ID":"52b8f3e2-445d-4854-8b3d-04b82e1e1473","Type":"ContainerStarted","Data":"2a956e9dcf453ada99b3d606087be9ba285beb59b5a89230896974d34c981080"}
Jul 22 02:30:55.528491 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:30:55.528440    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77d4f48d9d-fmxv9" podStartSLOduration=1.5284213850000001 podStartE2EDuration="1.528421385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:30:55.526703775 +0000 UTC m=+145.207436824" watchObservedRunningTime="2026-07-22 02:30:55.528421385 +0000 UTC m=+145.209154431"
Jul 22 02:31:04.601986 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:31:04.601944    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:31:04.602405 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:31:04.602018    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:31:04.606834 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:31:04.606809    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:31:05.545624 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:31:05.545596    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:32:52.514321 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.514239    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lkmbq"]
Jul 22 02:32:52.516234 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.516216    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lkmbq"
Jul 22 02:32:52.520802 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.520783    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gktrn\""
Jul 22 02:32:52.521276 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.521257    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 22 02:32:52.521478 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.521457    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 22 02:32:52.531831 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.531806    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lkmbq"]
Jul 22 02:32:52.586077 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.586015    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs2cv\" (UniqueName: \"kubernetes.io/projected/fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba-kube-api-access-qs2cv\") pod \"cert-manager-55b4d954d5-lkmbq\" (UID: \"fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba\") " pod="cert-manager/cert-manager-55b4d954d5-lkmbq"
Jul 22 02:32:52.586246 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.586105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba-bound-sa-token\") pod \"cert-manager-55b4d954d5-lkmbq\" (UID: \"fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba\") " pod="cert-manager/cert-manager-55b4d954d5-lkmbq"
Jul 22 02:32:52.687422 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.687386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba-bound-sa-token\") pod \"cert-manager-55b4d954d5-lkmbq\" (UID: \"fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba\") " pod="cert-manager/cert-manager-55b4d954d5-lkmbq"
Jul 22 02:32:52.687578 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.687474    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs2cv\" (UniqueName: \"kubernetes.io/projected/fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba-kube-api-access-qs2cv\") pod \"cert-manager-55b4d954d5-lkmbq\" (UID: \"fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba\") " pod="cert-manager/cert-manager-55b4d954d5-lkmbq"
Jul 22 02:32:52.696420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.696385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba-bound-sa-token\") pod \"cert-manager-55b4d954d5-lkmbq\" (UID: \"fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba\") " pod="cert-manager/cert-manager-55b4d954d5-lkmbq"
Jul 22 02:32:52.696672 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.696651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs2cv\" (UniqueName: \"kubernetes.io/projected/fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba-kube-api-access-qs2cv\") pod \"cert-manager-55b4d954d5-lkmbq\" (UID: \"fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba\") " pod="cert-manager/cert-manager-55b4d954d5-lkmbq"
Jul 22 02:32:52.832090 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.831976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-lkmbq"
Jul 22 02:32:52.958913 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:52.958883    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-lkmbq"]
Jul 22 02:32:52.961759 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:32:52.961734    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd2cd7f2_5bf1_4b2a_bd57_e836a4b165ba.slice/crio-75df38ec6a2cb0153b2bc570b7956d1ab217612c0b52f52e0dbd4a6f2ea12d04 WatchSource:0}: Error finding container 75df38ec6a2cb0153b2bc570b7956d1ab217612c0b52f52e0dbd4a6f2ea12d04: Status 404 returned error can't find the container with id 75df38ec6a2cb0153b2bc570b7956d1ab217612c0b52f52e0dbd4a6f2ea12d04
Jul 22 02:32:53.925203 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:53.925163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lkmbq" event={"ID":"fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba","Type":"ContainerStarted","Data":"75df38ec6a2cb0153b2bc570b7956d1ab217612c0b52f52e0dbd4a6f2ea12d04"}
Jul 22 02:32:58.946516 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:58.946473    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-lkmbq" event={"ID":"fd2cd7f2-5bf1-4b2a-bd57-e836a4b165ba","Type":"ContainerStarted","Data":"1022fee22fae84498e7e478588703d3e7bc376030a9a2e6aa387efbd482327df"}
Jul 22 02:32:58.969781 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:32:58.969735    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-lkmbq" podStartSLOduration=1.67704715 podStartE2EDuration="6.96972042s" totalImagesPullingTime="5.29267327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:32:58.968796768 +0000 UTC m=+268.649529813" watchObservedRunningTime="2026-07-22 02:32:58.96972042 +0000 UTC m=+268.650453466"
Jul 22 02:33:16.313990 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.313955    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"]
Jul 22 02:33:16.316256 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.316240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.319672 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.319650    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 22 02:33:16.320525 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.320502    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 22 02:33:16.320679 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.320511    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 22 02:33:16.320679 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.320583    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gwnxn\""
Jul 22 02:33:16.320940 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.320918    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 22 02:33:16.349303 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.349277    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"]
Jul 22 02:33:16.481616 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.481581    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/659b312c-abf5-4824-90e0-3786f049a3a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.481788 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.481620    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/659b312c-abf5-4824-90e0-3786f049a3a8-tmp\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.481788 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.481701    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/659b312c-abf5-4824-90e0-3786f049a3a8-manifests\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.481788 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.481766    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/659b312c-abf5-4824-90e0-3786f049a3a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.481905 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.481786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65rg\" (UniqueName: \"kubernetes.io/projected/659b312c-abf5-4824-90e0-3786f049a3a8-kube-api-access-n65rg\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.582933 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.582834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/659b312c-abf5-4824-90e0-3786f049a3a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.582933 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.582879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65rg\" (UniqueName: \"kubernetes.io/projected/659b312c-abf5-4824-90e0-3786f049a3a8-kube-api-access-n65rg\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.583161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.582940    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/659b312c-abf5-4824-90e0-3786f049a3a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.583161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.582968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/659b312c-abf5-4824-90e0-3786f049a3a8-tmp\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.583161 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.583010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/659b312c-abf5-4824-90e0-3786f049a3a8-manifests\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.583479 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.583453    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/659b312c-abf5-4824-90e0-3786f049a3a8-manifests\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.585410 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.585382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/659b312c-abf5-4824-90e0-3786f049a3a8-tmp\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.585540 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.585522    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/659b312c-abf5-4824-90e0-3786f049a3a8-webhook-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.585652 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.585635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/659b312c-abf5-4824-90e0-3786f049a3a8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.592098 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.592077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65rg\" (UniqueName: \"kubernetes.io/projected/659b312c-abf5-4824-90e0-3786f049a3a8-kube-api-access-n65rg\") pod \"opendatahub-operator-controller-manager-86c79b8765-q2zdv\" (UID: \"659b312c-abf5-4824-90e0-3786f049a3a8\") " pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.625813 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.625772    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:16.769217 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:16.769192    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"]
Jul 22 02:33:16.771614 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:33:16.771588    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659b312c_abf5_4824_90e0_3786f049a3a8.slice/crio-b8d37ea58a567adc45b1a9ca5f368e317602432f1babaa7e73a8ea080fca83be WatchSource:0}: Error finding container b8d37ea58a567adc45b1a9ca5f368e317602432f1babaa7e73a8ea080fca83be: Status 404 returned error can't find the container with id b8d37ea58a567adc45b1a9ca5f368e317602432f1babaa7e73a8ea080fca83be
Jul 22 02:33:17.011886 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:17.011851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv" event={"ID":"659b312c-abf5-4824-90e0-3786f049a3a8","Type":"ContainerStarted","Data":"b8d37ea58a567adc45b1a9ca5f368e317602432f1babaa7e73a8ea080fca83be"}
Jul 22 02:33:21.029476 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:21.029435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv" event={"ID":"659b312c-abf5-4824-90e0-3786f049a3a8","Type":"ContainerStarted","Data":"1077a90e458796c58760dcc50c9a41342fc9a2482b9d3456475f4d3b4fa87cb6"}
Jul 22 02:33:22.034574 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:22.034540    2583 generic.go:356] "Generic (PLEG): container finished" podID="659b312c-abf5-4824-90e0-3786f049a3a8" containerID="1077a90e458796c58760dcc50c9a41342fc9a2482b9d3456475f4d3b4fa87cb6" exitCode=0
Jul 22 02:33:22.034970 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:22.034592    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv" event={"ID":"659b312c-abf5-4824-90e0-3786f049a3a8","Type":"ContainerDied","Data":"1077a90e458796c58760dcc50c9a41342fc9a2482b9d3456475f4d3b4fa87cb6"}
Jul 22 02:33:23.039649 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:23.039611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv" event={"ID":"659b312c-abf5-4824-90e0-3786f049a3a8","Type":"ContainerStarted","Data":"6afc199212e3218d6dd3cbc08d8d05e684d07e9909f519127c96b157cfa047ef"}
Jul 22 02:33:23.040064 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:23.039742    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:23.061806 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:23.061755    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv" podStartSLOduration=2.305059034 podStartE2EDuration="7.061740079s" totalImagesPullingTime="3.730824505s" totalInitContainerRuntime="1.02585654s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:23.060395157 +0000 UTC m=+292.741128203" watchObservedRunningTime="2026-07-22 02:33:23.061740079 +0000 UTC m=+292.742473131"
Jul 22 02:33:30.698657 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:30.698630    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:33:30.700752 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:30.700732    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:33:30.703069 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:30.703050    2583 kubelet.go:1648] "Image garbage collection succeeded"
Jul 22 02:33:33.960422 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.960391    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"]
Jul 22 02:33:33.963076 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.963061    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:33.969573 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.969550    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 22 02:33:33.970467 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.970449    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 22 02:33:33.970559 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.970494    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 22 02:33:33.970607 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.970563    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 22 02:33:33.970654 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.970631    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 22 02:33:33.970697 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.970665    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nlphm\""
Jul 22 02:33:33.983722 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:33.983700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"]
Jul 22 02:33:34.020134 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.020102    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-manager-config\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.020300 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.020150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssqks\" (UniqueName: \"kubernetes.io/projected/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-kube-api-access-ssqks\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.020300 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.020168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-metrics-cert\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.020300 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.020266    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-cert\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.046554 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.046526    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86c79b8765-q2zdv"
Jul 22 02:33:34.121545 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.121515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssqks\" (UniqueName: \"kubernetes.io/projected/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-kube-api-access-ssqks\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.121720 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.121552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-metrics-cert\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.121787 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.121733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-cert\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.121861 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.121842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-manager-config\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.122496 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.122453    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-manager-config\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.124270 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.124245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-metrics-cert\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.124464 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.124445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-cert\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.131708 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.131683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssqks\" (UniqueName: \"kubernetes.io/projected/629f68b7-5d4e-4bc3-b58c-a0ad98348a76-kube-api-access-ssqks\") pod \"lws-controller-manager-57d76bdcd7-qr4bm\" (UID: \"629f68b7-5d4e-4bc3-b58c-a0ad98348a76\") " pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.271918 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.271880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:34.408086 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.408062    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"]
Jul 22 02:33:34.409809 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:33:34.409780    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629f68b7_5d4e_4bc3_b58c_a0ad98348a76.slice/crio-78826512b727cf99bdc838da7dc39340c7d953aa698b4b5001281ebda0bcc789 WatchSource:0}: Error finding container 78826512b727cf99bdc838da7dc39340c7d953aa698b4b5001281ebda0bcc789: Status 404 returned error can't find the container with id 78826512b727cf99bdc838da7dc39340c7d953aa698b4b5001281ebda0bcc789
Jul 22 02:33:34.411690 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:34.411676    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:33:35.085763 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:35.085732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm" event={"ID":"629f68b7-5d4e-4bc3-b58c-a0ad98348a76","Type":"ContainerStarted","Data":"78826512b727cf99bdc838da7dc39340c7d953aa698b4b5001281ebda0bcc789"}
Jul 22 02:33:36.618045 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.617992    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"]
Jul 22 02:33:36.625463 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.625438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.628871 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.628845    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 22 02:33:36.629414 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.629192    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h2gxf\""
Jul 22 02:33:36.629722 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.629703    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 22 02:33:36.634582 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.634559    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 22 02:33:36.642253 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.642228    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 22 02:33:36.653735 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.653701    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"]
Jul 22 02:33:36.744861 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.744827    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a460d2a8-9522-4b15-9997-064d5005bf0c-tls-certs\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.745053 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.744880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsvxc\" (UniqueName: \"kubernetes.io/projected/a460d2a8-9522-4b15-9997-064d5005bf0c-kube-api-access-dsvxc\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.745053 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.744982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a460d2a8-9522-4b15-9997-064d5005bf0c-tmp\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.845782 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.845743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a460d2a8-9522-4b15-9997-064d5005bf0c-tls-certs\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.845782 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.845787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsvxc\" (UniqueName: \"kubernetes.io/projected/a460d2a8-9522-4b15-9997-064d5005bf0c-kube-api-access-dsvxc\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.846058 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.845816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a460d2a8-9522-4b15-9997-064d5005bf0c-tmp\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.848422 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.848396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a460d2a8-9522-4b15-9997-064d5005bf0c-tmp\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.848698 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.848670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a460d2a8-9522-4b15-9997-064d5005bf0c-tls-certs\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.862248 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.862227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsvxc\" (UniqueName: \"kubernetes.io/projected/a460d2a8-9522-4b15-9997-064d5005bf0c-kube-api-access-dsvxc\") pod \"kube-auth-proxy-7fd555b8c5-q4pjh\" (UID: \"a460d2a8-9522-4b15-9997-064d5005bf0c\") " pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:36.937839 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:36.937756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"
Jul 22 02:33:37.229182 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:37.229146    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh"]
Jul 22 02:33:37.232622 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:33:37.232595    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda460d2a8_9522_4b15_9997_064d5005bf0c.slice/crio-c392a3014740819bece15c111249840c2787ff57c386917bd042e7d8896c839b WatchSource:0}: Error finding container c392a3014740819bece15c111249840c2787ff57c386917bd042e7d8896c839b: Status 404 returned error can't find the container with id c392a3014740819bece15c111249840c2787ff57c386917bd042e7d8896c839b
Jul 22 02:33:38.102084 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:38.102020    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm" event={"ID":"629f68b7-5d4e-4bc3-b58c-a0ad98348a76","Type":"ContainerStarted","Data":"1f24c7a9c1daf10c67ccd78ed7010e53de56560dc023c8b7f75a092f1f13b565"}
Jul 22 02:33:38.102559 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:38.102111    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:38.103286 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:38.103260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh" event={"ID":"a460d2a8-9522-4b15-9997-064d5005bf0c","Type":"ContainerStarted","Data":"c392a3014740819bece15c111249840c2787ff57c386917bd042e7d8896c839b"}
Jul 22 02:33:40.113617 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:40.113583    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh" event={"ID":"a460d2a8-9522-4b15-9997-064d5005bf0c","Type":"ContainerStarted","Data":"0a3d35cc60f84bb536c0c6a339aa513fa62183f4eef2a3f39d0a71bafc157805"}
Jul 22 02:33:40.196657 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:40.196586    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7fd555b8c5-q4pjh" podStartSLOduration=1.908291716 podStartE2EDuration="4.196567248s" totalImagesPullingTime="2.288275532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:40.195530539 +0000 UTC m=+309.876263585" watchObservedRunningTime="2026-07-22 02:33:40.196567248 +0000 UTC m=+309.877300294"
Jul 22 02:33:40.197686 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:40.197649    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm" podStartSLOduration=4.486997451 podStartE2EDuration="7.197636363s" totalImagesPullingTime="2.710638912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:38.144862405 +0000 UTC m=+307.825595453" watchObservedRunningTime="2026-07-22 02:33:40.197636363 +0000 UTC m=+309.878369409"
Jul 22 02:33:49.110799 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.110764    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57d76bdcd7-qr4bm"
Jul 22 02:33:49.345726 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.345692    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"]
Jul 22 02:33:49.349170 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.349153    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:49.354270 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.354250    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-spmzr\""
Jul 22 02:33:49.354871 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.354697    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 22 02:33:49.365742 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.365677    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"]
Jul 22 02:33:49.460667 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.460627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6df209-b6e5-4aa0-aef3-543b387a741e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l5pnp\" (UID: \"cd6df209-b6e5-4aa0-aef3-543b387a741e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:49.460855 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.460688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppd8s\" (UniqueName: \"kubernetes.io/projected/cd6df209-b6e5-4aa0-aef3-543b387a741e-kube-api-access-ppd8s\") pod \"llmisvc-controller-manager-777c6bd7d5-l5pnp\" (UID: \"cd6df209-b6e5-4aa0-aef3-543b387a741e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:49.561554 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.561512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6df209-b6e5-4aa0-aef3-543b387a741e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l5pnp\" (UID: \"cd6df209-b6e5-4aa0-aef3-543b387a741e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:49.561736 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.561584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppd8s\" (UniqueName: \"kubernetes.io/projected/cd6df209-b6e5-4aa0-aef3-543b387a741e-kube-api-access-ppd8s\") pod \"llmisvc-controller-manager-777c6bd7d5-l5pnp\" (UID: \"cd6df209-b6e5-4aa0-aef3-543b387a741e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:49.564089 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.564060    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6df209-b6e5-4aa0-aef3-543b387a741e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l5pnp\" (UID: \"cd6df209-b6e5-4aa0-aef3-543b387a741e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:49.575398 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.575374    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppd8s\" (UniqueName: \"kubernetes.io/projected/cd6df209-b6e5-4aa0-aef3-543b387a741e-kube-api-access-ppd8s\") pod \"llmisvc-controller-manager-777c6bd7d5-l5pnp\" (UID: \"cd6df209-b6e5-4aa0-aef3-543b387a741e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:49.659392 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.659300    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:49.857489 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:49.857454    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"]
Jul 22 02:33:49.860557 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:33:49.860516    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcd6df209_b6e5_4aa0_aef3_543b387a741e.slice/crio-daefbe91729664a35e70b6f6dd9e81c13bebd0cb5cf589602157ca5c504a15cf WatchSource:0}: Error finding container daefbe91729664a35e70b6f6dd9e81c13bebd0cb5cf589602157ca5c504a15cf: Status 404 returned error can't find the container with id daefbe91729664a35e70b6f6dd9e81c13bebd0cb5cf589602157ca5c504a15cf
Jul 22 02:33:50.158512 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:50.158469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp" event={"ID":"cd6df209-b6e5-4aa0-aef3-543b387a741e","Type":"ContainerStarted","Data":"daefbe91729664a35e70b6f6dd9e81c13bebd0cb5cf589602157ca5c504a15cf"}
Jul 22 02:33:53.173104 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:53.173063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp" event={"ID":"cd6df209-b6e5-4aa0-aef3-543b387a741e","Type":"ContainerStarted","Data":"c84c363fbbf33370e4f48fec8e4c7bcefc00347b15a428713960380742428f3b"}
Jul 22 02:33:53.173505 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:53.173154    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:33:53.211686 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:33:53.211633    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp" podStartSLOduration=1.079447173 podStartE2EDuration="4.2116182s" totalImagesPullingTime="3.132171027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:33:53.206923767 +0000 UTC m=+322.887656813" watchObservedRunningTime="2026-07-22 02:33:53.2116182 +0000 UTC m=+322.892351245"
Jul 22 02:34:24.180879 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:34:24.180796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l5pnp"
Jul 22 02:35:00.868541 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:00.868509    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d4f48d9d-fmxv9"]
Jul 22 02:35:25.888900 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:25.888860    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77d4f48d9d-fmxv9" podUID="52b8f3e2-445d-4854-8b3d-04b82e1e1473" containerName="console" containerID="cri-o://eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378" gracePeriod=15
Jul 22 02:35:26.132967 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.132942    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:35:26.238267 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238235    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-service-ca\") pod \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") "
Jul 22 02:35:26.238429 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238285    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-oauth-config\") pod \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") "
Jul 22 02:35:26.238429 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238305    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-trusted-ca-bundle\") pod \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") "
Jul 22 02:35:26.238429 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238356    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4sx8\" (UniqueName: \"kubernetes.io/projected/52b8f3e2-445d-4854-8b3d-04b82e1e1473-kube-api-access-z4sx8\") pod \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") "
Jul 22 02:35:26.238429 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238375    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-oauth-serving-cert\") pod \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") "
Jul 22 02:35:26.238429 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238401    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-serving-cert\") pod \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") "
Jul 22 02:35:26.238665 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238443    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-config\") pod \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\" (UID: \"52b8f3e2-445d-4854-8b3d-04b82e1e1473\") "
Jul 22 02:35:26.238765 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238738    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-service-ca" (OuterVolumeSpecName: "service-ca") pod "52b8f3e2-445d-4854-8b3d-04b82e1e1473" (UID: "52b8f3e2-445d-4854-8b3d-04b82e1e1473"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:35:26.238925 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238899    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "52b8f3e2-445d-4854-8b3d-04b82e1e1473" (UID: "52b8f3e2-445d-4854-8b3d-04b82e1e1473"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:35:26.238925 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "52b8f3e2-445d-4854-8b3d-04b82e1e1473" (UID: "52b8f3e2-445d-4854-8b3d-04b82e1e1473"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:35:26.239311 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.238991    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-config" (OuterVolumeSpecName: "console-config") pod "52b8f3e2-445d-4854-8b3d-04b82e1e1473" (UID: "52b8f3e2-445d-4854-8b3d-04b82e1e1473"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:35:26.240671 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.240643    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "52b8f3e2-445d-4854-8b3d-04b82e1e1473" (UID: "52b8f3e2-445d-4854-8b3d-04b82e1e1473"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:35:26.240671 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.240657    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b8f3e2-445d-4854-8b3d-04b82e1e1473-kube-api-access-z4sx8" (OuterVolumeSpecName: "kube-api-access-z4sx8") pod "52b8f3e2-445d-4854-8b3d-04b82e1e1473" (UID: "52b8f3e2-445d-4854-8b3d-04b82e1e1473"). InnerVolumeSpecName "kube-api-access-z4sx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:35:26.240795 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.240713    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "52b8f3e2-445d-4854-8b3d-04b82e1e1473" (UID: "52b8f3e2-445d-4854-8b3d-04b82e1e1473"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:35:26.339334 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.339296    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:35:26.339334 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.339328    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-oauth-config\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:35:26.339334 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.339339    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-trusted-ca-bundle\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:35:26.339552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.339349    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4sx8\" (UniqueName: \"kubernetes.io/projected/52b8f3e2-445d-4854-8b3d-04b82e1e1473-kube-api-access-z4sx8\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:35:26.339552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.339358    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-oauth-serving-cert\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:35:26.339552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.339366    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-serving-cert\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:35:26.339552 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.339374    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52b8f3e2-445d-4854-8b3d-04b82e1e1473-console-config\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:35:26.555779 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.555699    2583 generic.go:356] "Generic (PLEG): container finished" podID="52b8f3e2-445d-4854-8b3d-04b82e1e1473" containerID="eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378" exitCode=0
Jul 22 02:35:26.555779 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.555752    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d4f48d9d-fmxv9"
Jul 22 02:35:26.555779 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.555768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4f48d9d-fmxv9" event={"ID":"52b8f3e2-445d-4854-8b3d-04b82e1e1473","Type":"ContainerDied","Data":"eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378"}
Jul 22 02:35:26.556001 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.555808    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d4f48d9d-fmxv9" event={"ID":"52b8f3e2-445d-4854-8b3d-04b82e1e1473","Type":"ContainerDied","Data":"2a956e9dcf453ada99b3d606087be9ba285beb59b5a89230896974d34c981080"}
Jul 22 02:35:26.556001 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.555827    2583 scope.go:117] "RemoveContainer" containerID="eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378"
Jul 22 02:35:26.565528 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.565507    2583 scope.go:117] "RemoveContainer" containerID="eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378"
Jul 22 02:35:26.565828 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:35:26.565813    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378\": container with ID starting with eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378 not found: ID does not exist" containerID="eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378"
Jul 22 02:35:26.565872 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.565837    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378"} err="failed to get container status \"eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378\": rpc error: code = NotFound desc = could not find container \"eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378\": container with ID starting with eae1c50cda984c44031cadc25c79513a81e2fc37faad56d931b8bb7f2e4ad378 not found: ID does not exist"
Jul 22 02:35:26.584235 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.584208    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d4f48d9d-fmxv9"]
Jul 22 02:35:26.592609 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.592583    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77d4f48d9d-fmxv9"]
Jul 22 02:35:26.868186 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:26.868110    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52b8f3e2-445d-4854-8b3d-04b82e1e1473" path="/var/lib/kubelet/pods/52b8f3e2-445d-4854-8b3d-04b82e1e1473/volumes"
Jul 22 02:35:35.695537 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.695502    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"]
Jul 22 02:35:35.696069 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.695793    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52b8f3e2-445d-4854-8b3d-04b82e1e1473" containerName="console"
Jul 22 02:35:35.696069 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.695803    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="52b8f3e2-445d-4854-8b3d-04b82e1e1473" containerName="console"
Jul 22 02:35:35.696069 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.695873    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="52b8f3e2-445d-4854-8b3d-04b82e1e1473" containerName="console"
Jul 22 02:35:35.698927 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.698909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.703437 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.703412    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 22 02:35:35.704162 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.704131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 22 02:35:35.704285 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.704131    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 22 02:35:35.704285 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.704170    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 22 02:35:35.704285 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.704131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-srhxs\""
Jul 22 02:35:35.723407 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.722586    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"]
Jul 22 02:35:35.820951 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.820912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/db88cc7f-7510-4c8c-96b1-2175a4e706b9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.820951 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.820967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db88cc7f-7510-4c8c-96b1-2175a4e706b9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.821241 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.820989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcgmw\" (UniqueName: \"kubernetes.io/projected/db88cc7f-7510-4c8c-96b1-2175a4e706b9-kube-api-access-lcgmw\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.921867 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.921833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/db88cc7f-7510-4c8c-96b1-2175a4e706b9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.922057 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.921888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db88cc7f-7510-4c8c-96b1-2175a4e706b9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.922057 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.921922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcgmw\" (UniqueName: \"kubernetes.io/projected/db88cc7f-7510-4c8c-96b1-2175a4e706b9-kube-api-access-lcgmw\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.922601 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.922575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/db88cc7f-7510-4c8c-96b1-2175a4e706b9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.924369 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.924346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/db88cc7f-7510-4c8c-96b1-2175a4e706b9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:35.931237 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:35.931211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcgmw\" (UniqueName: \"kubernetes.io/projected/db88cc7f-7510-4c8c-96b1-2175a4e706b9-kube-api-access-lcgmw\") pod \"kuadrant-console-plugin-66b5774985-52ptn\" (UID: \"db88cc7f-7510-4c8c-96b1-2175a4e706b9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:36.008159 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:36.008122    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"
Jul 22 02:35:36.140484 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:36.140445    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn"]
Jul 22 02:35:36.144082 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:35:36.144023    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb88cc7f_7510_4c8c_96b1_2175a4e706b9.slice/crio-291ce1b4474d3f5732807228f21f6801e6f8a998af247101c5fd9d667a770b1a WatchSource:0}: Error finding container 291ce1b4474d3f5732807228f21f6801e6f8a998af247101c5fd9d667a770b1a: Status 404 returned error can't find the container with id 291ce1b4474d3f5732807228f21f6801e6f8a998af247101c5fd9d667a770b1a
Jul 22 02:35:36.602835 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:36.602797    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn" event={"ID":"db88cc7f-7510-4c8c-96b1-2175a4e706b9","Type":"ContainerStarted","Data":"291ce1b4474d3f5732807228f21f6801e6f8a998af247101c5fd9d667a770b1a"}
Jul 22 02:35:45.928862 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:45.928830    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"]
Jul 22 02:35:45.954079 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:45.954017    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"]
Jul 22 02:35:45.954241 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:45.954180    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:35:45.957281 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:45.957251    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-d4w27\""
Jul 22 02:35:46.026793 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:46.026759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dfxb\" (UniqueName: \"kubernetes.io/projected/124ca64f-a7d4-4b05-b5c0-678abca1e3da-kube-api-access-6dfxb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6bww5\" (UID: \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:35:46.026985 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:46.026837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/124ca64f-a7d4-4b05-b5c0-678abca1e3da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6bww5\" (UID: \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:35:46.127844 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:46.127795    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dfxb\" (UniqueName: \"kubernetes.io/projected/124ca64f-a7d4-4b05-b5c0-678abca1e3da-kube-api-access-6dfxb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6bww5\" (UID: \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:35:46.128056 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:46.127887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/124ca64f-a7d4-4b05-b5c0-678abca1e3da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6bww5\" (UID: \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:35:46.128362 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:46.128337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/124ca64f-a7d4-4b05-b5c0-678abca1e3da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6bww5\" (UID: \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:35:46.140638 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:46.140612    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dfxb\" (UniqueName: \"kubernetes.io/projected/124ca64f-a7d4-4b05-b5c0-678abca1e3da-kube-api-access-6dfxb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6bww5\" (UID: \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:35:46.267749 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:46.267707    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:35:57.748403 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:57.748373    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"]
Jul 22 02:35:57.751652 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:35:57.751623    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124ca64f_a7d4_4b05_b5c0_678abca1e3da.slice/crio-343548e1cce60b2cbb891ccf5ff13693928324fc836fc61163b76e093c76bbb8 WatchSource:0}: Error finding container 343548e1cce60b2cbb891ccf5ff13693928324fc836fc61163b76e093c76bbb8: Status 404 returned error can't find the container with id 343548e1cce60b2cbb891ccf5ff13693928324fc836fc61163b76e093c76bbb8
Jul 22 02:35:58.158331 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:58.158296    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"]
Jul 22 02:35:58.713914 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:58.713869    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn" event={"ID":"db88cc7f-7510-4c8c-96b1-2175a4e706b9","Type":"ContainerStarted","Data":"78d45f7ffae9ddd689c4209628ffbbdc496b0e3df11892cabbd7734f15757d68"}
Jul 22 02:35:58.715375 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:58.715342    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5" event={"ID":"124ca64f-a7d4-4b05-b5c0-678abca1e3da","Type":"ContainerStarted","Data":"343548e1cce60b2cbb891ccf5ff13693928324fc836fc61163b76e093c76bbb8"}
Jul 22 02:35:58.735872 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:35:58.735811    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-52ptn" podStartSLOduration=2.181562083 podStartE2EDuration="23.735791474s" totalImagesPullingTime="21.554229391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:35:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:35:58.734743037 +0000 UTC m=+448.415476085" watchObservedRunningTime="2026-07-22 02:35:58.735791474 +0000 UTC m=+448.416524522"
Jul 22 02:36:02.736147 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.736111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5" event={"ID":"124ca64f-a7d4-4b05-b5c0-678abca1e3da","Type":"ContainerStarted","Data":"7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed"}
Jul 22 02:36:02.736577 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.736225    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5" podUID="124ca64f-a7d4-4b05-b5c0-678abca1e3da" containerName="manager" containerID="cri-o://7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed" gracePeriod=10
Jul 22 02:36:02.736577 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.736279    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:36:02.758475 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.758425    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5" podStartSLOduration=13.51731426 podStartE2EDuration="17.758408311s" totalImagesPullingTime="4.241094051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:36:02.757633533 +0000 UTC m=+452.438366578" watchObservedRunningTime="2026-07-22 02:36:02.758408311 +0000 UTC m=+452.439141424"
Jul 22 02:36:02.978441 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.978407    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:36:02.986686 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.986627    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/124ca64f-a7d4-4b05-b5c0-678abca1e3da-extensions-socket-volume\") pod \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\" (UID: \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\") "
Jul 22 02:36:02.986686 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.986676    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dfxb\" (UniqueName: \"kubernetes.io/projected/124ca64f-a7d4-4b05-b5c0-678abca1e3da-kube-api-access-6dfxb\") pod \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\" (UID: \"124ca64f-a7d4-4b05-b5c0-678abca1e3da\") "
Jul 22 02:36:02.987110 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.987085    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124ca64f-a7d4-4b05-b5c0-678abca1e3da-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "124ca64f-a7d4-4b05-b5c0-678abca1e3da" (UID: "124ca64f-a7d4-4b05-b5c0-678abca1e3da"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:36:02.988903 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:02.988883    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124ca64f-a7d4-4b05-b5c0-678abca1e3da-kube-api-access-6dfxb" (OuterVolumeSpecName: "kube-api-access-6dfxb") pod "124ca64f-a7d4-4b05-b5c0-678abca1e3da" (UID: "124ca64f-a7d4-4b05-b5c0-678abca1e3da"). InnerVolumeSpecName "kube-api-access-6dfxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:36:03.087261 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.087222    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/124ca64f-a7d4-4b05-b5c0-678abca1e3da-extensions-socket-volume\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:36:03.087261 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.087257    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dfxb\" (UniqueName: \"kubernetes.io/projected/124ca64f-a7d4-4b05-b5c0-678abca1e3da-kube-api-access-6dfxb\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:36:03.741257 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.741224    2583 generic.go:356] "Generic (PLEG): container finished" podID="124ca64f-a7d4-4b05-b5c0-678abca1e3da" containerID="7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed" exitCode=0
Jul 22 02:36:03.741664 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.741272    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"
Jul 22 02:36:03.741664 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.741278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5" event={"ID":"124ca64f-a7d4-4b05-b5c0-678abca1e3da","Type":"ContainerDied","Data":"7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed"}
Jul 22 02:36:03.741664 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.741318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5" event={"ID":"124ca64f-a7d4-4b05-b5c0-678abca1e3da","Type":"ContainerDied","Data":"343548e1cce60b2cbb891ccf5ff13693928324fc836fc61163b76e093c76bbb8"}
Jul 22 02:36:03.741664 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.741339    2583 scope.go:117] "RemoveContainer" containerID="7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed"
Jul 22 02:36:03.750116 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.750098    2583 scope.go:117] "RemoveContainer" containerID="7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed"
Jul 22 02:36:03.750364 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:36:03.750346    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed\": container with ID starting with 7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed not found: ID does not exist" containerID="7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed"
Jul 22 02:36:03.750409 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.750375    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed"} err="failed to get container status \"7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed\": rpc error: code = NotFound desc = could not find container \"7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed\": container with ID starting with 7abf427a713ff9fb012564d981255c368efab041ec45f779a029b843c0be81ed not found: ID does not exist"
Jul 22 02:36:03.762421 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.762396    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"]
Jul 22 02:36:03.774911 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:03.774885    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6bww5"]
Jul 22 02:36:04.868039 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:04.867990    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124ca64f-a7d4-4b05-b5c0-678abca1e3da" path="/var/lib/kubelet/pods/124ca64f-a7d4-4b05-b5c0-678abca1e3da/volumes"
Jul 22 02:36:27.929826 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:27.929788    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2482"]
Jul 22 02:36:27.930302 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:27.930084    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124ca64f-a7d4-4b05-b5c0-678abca1e3da" containerName="manager"
Jul 22 02:36:27.930302 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:27.930095    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="124ca64f-a7d4-4b05-b5c0-678abca1e3da" containerName="manager"
Jul 22 02:36:27.930302 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:27.930158    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="124ca64f-a7d4-4b05-b5c0-678abca1e3da" containerName="manager"
Jul 22 02:36:27.994514 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:27.994484    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2482"]
Jul 22 02:36:27.994662 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:27.994591    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:27.997687 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:27.997661    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 22 02:36:28.038586 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.038554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2482"]
Jul 22 02:36:28.103450 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.103414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrnff\" (UniqueName: \"kubernetes.io/projected/0515ea8b-63c5-462e-b04e-1a467c5e86d4-kube-api-access-vrnff\") pod \"limitador-limitador-64c94fd64f-x2482\" (UID: \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:28.103623 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.103472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0515ea8b-63c5-462e-b04e-1a467c5e86d4-config-file\") pod \"limitador-limitador-64c94fd64f-x2482\" (UID: \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:28.204344 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.204265    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0515ea8b-63c5-462e-b04e-1a467c5e86d4-config-file\") pod \"limitador-limitador-64c94fd64f-x2482\" (UID: \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:28.204496 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.204342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrnff\" (UniqueName: \"kubernetes.io/projected/0515ea8b-63c5-462e-b04e-1a467c5e86d4-kube-api-access-vrnff\") pod \"limitador-limitador-64c94fd64f-x2482\" (UID: \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:28.204873 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.204853    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0515ea8b-63c5-462e-b04e-1a467c5e86d4-config-file\") pod \"limitador-limitador-64c94fd64f-x2482\" (UID: \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:28.228000 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.227965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrnff\" (UniqueName: \"kubernetes.io/projected/0515ea8b-63c5-462e-b04e-1a467c5e86d4-kube-api-access-vrnff\") pod \"limitador-limitador-64c94fd64f-x2482\" (UID: \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:28.305159 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.305121    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:28.433961 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.433927    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2482"]
Jul 22 02:36:28.436940 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:36:28.436909    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0515ea8b_63c5_462e_b04e_1a467c5e86d4.slice/crio-4154d6792a466183995dd010282e16b08093f1bc3de169a8e4c876b98f77ef46 WatchSource:0}: Error finding container 4154d6792a466183995dd010282e16b08093f1bc3de169a8e4c876b98f77ef46: Status 404 returned error can't find the container with id 4154d6792a466183995dd010282e16b08093f1bc3de169a8e4c876b98f77ef46
Jul 22 02:36:28.726189 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.726154    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-whssf"]
Jul 22 02:36:28.728957 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.728941    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-whssf"
Jul 22 02:36:28.731431 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.731411    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gfzxx\""
Jul 22 02:36:28.740453 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.740432    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-whssf"]
Jul 22 02:36:28.809592 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.809553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68dc\" (UniqueName: \"kubernetes.io/projected/5c44f7f7-59f9-4cdc-8081-b04a63a82f89-kube-api-access-t68dc\") pod \"authorino-58cd955675-whssf\" (UID: \"5c44f7f7-59f9-4cdc-8081-b04a63a82f89\") " pod="kuadrant-system/authorino-58cd955675-whssf"
Jul 22 02:36:28.845096 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.845064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482" event={"ID":"0515ea8b-63c5-462e-b04e-1a467c5e86d4","Type":"ContainerStarted","Data":"4154d6792a466183995dd010282e16b08093f1bc3de169a8e4c876b98f77ef46"}
Jul 22 02:36:28.910397 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.910364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68dc\" (UniqueName: \"kubernetes.io/projected/5c44f7f7-59f9-4cdc-8081-b04a63a82f89-kube-api-access-t68dc\") pod \"authorino-58cd955675-whssf\" (UID: \"5c44f7f7-59f9-4cdc-8081-b04a63a82f89\") " pod="kuadrant-system/authorino-58cd955675-whssf"
Jul 22 02:36:28.920101 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:28.920068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68dc\" (UniqueName: \"kubernetes.io/projected/5c44f7f7-59f9-4cdc-8081-b04a63a82f89-kube-api-access-t68dc\") pod \"authorino-58cd955675-whssf\" (UID: \"5c44f7f7-59f9-4cdc-8081-b04a63a82f89\") " pod="kuadrant-system/authorino-58cd955675-whssf"
Jul 22 02:36:29.038441 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:29.038406    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-whssf"
Jul 22 02:36:29.188288 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:29.188254    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-whssf"]
Jul 22 02:36:29.192499 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:36:29.192464    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c44f7f7_59f9_4cdc_8081_b04a63a82f89.slice/crio-3479a1d87704ba57f98cc3665cb1ee4392d59168b14d5c482d93de07fa3d291b WatchSource:0}: Error finding container 3479a1d87704ba57f98cc3665cb1ee4392d59168b14d5c482d93de07fa3d291b: Status 404 returned error can't find the container with id 3479a1d87704ba57f98cc3665cb1ee4392d59168b14d5c482d93de07fa3d291b
Jul 22 02:36:29.854730 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:29.854611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-whssf" event={"ID":"5c44f7f7-59f9-4cdc-8081-b04a63a82f89","Type":"ContainerStarted","Data":"3479a1d87704ba57f98cc3665cb1ee4392d59168b14d5c482d93de07fa3d291b"}
Jul 22 02:36:30.719441 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:30.719416    2583 scope.go:117] "RemoveContainer" containerID="a64fadb563aa926dfd633e7098826af58e07b6fae4acac9fd6910114822bd404"
Jul 22 02:36:32.396139 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:32.396102    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-whssf"]
Jul 22 02:36:33.876086 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:33.876017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-whssf" event={"ID":"5c44f7f7-59f9-4cdc-8081-b04a63a82f89","Type":"ContainerStarted","Data":"449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2"}
Jul 22 02:36:33.876086 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:33.876060    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-whssf" podUID="5c44f7f7-59f9-4cdc-8081-b04a63a82f89" containerName="authorino" containerID="cri-o://449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2" gracePeriod=30
Jul 22 02:36:33.877743 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:33.877716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482" event={"ID":"0515ea8b-63c5-462e-b04e-1a467c5e86d4","Type":"ContainerStarted","Data":"48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26"}
Jul 22 02:36:33.877875 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:33.877861    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:33.892291 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:33.892249    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-whssf" podStartSLOduration=2.000815762 podStartE2EDuration="5.892234979s" totalImagesPullingTime="3.891419217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:36:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:36:33.890664244 +0000 UTC m=+483.571397291" watchObservedRunningTime="2026-07-22 02:36:33.892234979 +0000 UTC m=+483.572968025"
Jul 22 02:36:33.906774 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:33.906730    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482" podStartSLOduration=2.22167528 podStartE2EDuration="6.906715143s" totalImagesPullingTime="4.685039863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:36:33.9055936 +0000 UTC m=+483.586326648" watchObservedRunningTime="2026-07-22 02:36:33.906715143 +0000 UTC m=+483.587448189"
Jul 22 02:36:34.124407 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.124384    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-whssf"
Jul 22 02:36:34.265542 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.265506    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t68dc\" (UniqueName: \"kubernetes.io/projected/5c44f7f7-59f9-4cdc-8081-b04a63a82f89-kube-api-access-t68dc\") pod \"5c44f7f7-59f9-4cdc-8081-b04a63a82f89\" (UID: \"5c44f7f7-59f9-4cdc-8081-b04a63a82f89\") "
Jul 22 02:36:34.267690 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.267659    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c44f7f7-59f9-4cdc-8081-b04a63a82f89-kube-api-access-t68dc" (OuterVolumeSpecName: "kube-api-access-t68dc") pod "5c44f7f7-59f9-4cdc-8081-b04a63a82f89" (UID: "5c44f7f7-59f9-4cdc-8081-b04a63a82f89"). InnerVolumeSpecName "kube-api-access-t68dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:36:34.367011 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.366957    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t68dc\" (UniqueName: \"kubernetes.io/projected/5c44f7f7-59f9-4cdc-8081-b04a63a82f89-kube-api-access-t68dc\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:36:34.883120 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.883089    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c44f7f7-59f9-4cdc-8081-b04a63a82f89" containerID="449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2" exitCode=0
Jul 22 02:36:34.883489 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.883127    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-whssf"
Jul 22 02:36:34.883489 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.883170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-whssf" event={"ID":"5c44f7f7-59f9-4cdc-8081-b04a63a82f89","Type":"ContainerDied","Data":"449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2"}
Jul 22 02:36:34.883489 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.883205    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-whssf" event={"ID":"5c44f7f7-59f9-4cdc-8081-b04a63a82f89","Type":"ContainerDied","Data":"3479a1d87704ba57f98cc3665cb1ee4392d59168b14d5c482d93de07fa3d291b"}
Jul 22 02:36:34.883489 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.883221    2583 scope.go:117] "RemoveContainer" containerID="449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2"
Jul 22 02:36:34.891179 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.891161    2583 scope.go:117] "RemoveContainer" containerID="449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2"
Jul 22 02:36:34.891422 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:36:34.891403    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2\": container with ID starting with 449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2 not found: ID does not exist" containerID="449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2"
Jul 22 02:36:34.891481 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.891433    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2"} err="failed to get container status \"449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2\": rpc error: code = NotFound desc = could not find container \"449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2\": container with ID starting with 449a92b9138e5ef2e77e1fb53630e056c23035f48a29e215f4e82c0bae664cb2 not found: ID does not exist"
Jul 22 02:36:34.901380 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.901356    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-whssf"]
Jul 22 02:36:34.905781 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:34.905759    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-whssf"]
Jul 22 02:36:36.870472 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:36.870431    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c44f7f7-59f9-4cdc-8081-b04a63a82f89" path="/var/lib/kubelet/pods/5c44f7f7-59f9-4cdc-8081-b04a63a82f89/volumes"
Jul 22 02:36:42.622414 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:42.622377    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2482"]
Jul 22 02:36:42.622827 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:42.622565    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482" podUID="0515ea8b-63c5-462e-b04e-1a467c5e86d4" containerName="limitador" containerID="cri-o://48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26" gracePeriod=30
Jul 22 02:36:42.623292 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:42.623270    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:43.170139 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.170112    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:43.342124 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.342091    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0515ea8b-63c5-462e-b04e-1a467c5e86d4-config-file\") pod \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\" (UID: \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\") "
Jul 22 02:36:43.342289 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.342133    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrnff\" (UniqueName: \"kubernetes.io/projected/0515ea8b-63c5-462e-b04e-1a467c5e86d4-kube-api-access-vrnff\") pod \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\" (UID: \"0515ea8b-63c5-462e-b04e-1a467c5e86d4\") "
Jul 22 02:36:43.342459 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.342436    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0515ea8b-63c5-462e-b04e-1a467c5e86d4-config-file" (OuterVolumeSpecName: "config-file") pod "0515ea8b-63c5-462e-b04e-1a467c5e86d4" (UID: "0515ea8b-63c5-462e-b04e-1a467c5e86d4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:36:43.344399 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.344381    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0515ea8b-63c5-462e-b04e-1a467c5e86d4-kube-api-access-vrnff" (OuterVolumeSpecName: "kube-api-access-vrnff") pod "0515ea8b-63c5-462e-b04e-1a467c5e86d4" (UID: "0515ea8b-63c5-462e-b04e-1a467c5e86d4"). InnerVolumeSpecName "kube-api-access-vrnff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:36:43.442886 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.442843    2583 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0515ea8b-63c5-462e-b04e-1a467c5e86d4-config-file\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:36:43.442886 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.442876    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrnff\" (UniqueName: \"kubernetes.io/projected/0515ea8b-63c5-462e-b04e-1a467c5e86d4-kube-api-access-vrnff\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:36:43.918225 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.918186    2583 generic.go:356] "Generic (PLEG): container finished" podID="0515ea8b-63c5-462e-b04e-1a467c5e86d4" containerID="48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26" exitCode=0
Jul 22 02:36:43.918588 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.918233    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482"
Jul 22 02:36:43.918588 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.918276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482" event={"ID":"0515ea8b-63c5-462e-b04e-1a467c5e86d4","Type":"ContainerDied","Data":"48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26"}
Jul 22 02:36:43.918588 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.918334    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-x2482" event={"ID":"0515ea8b-63c5-462e-b04e-1a467c5e86d4","Type":"ContainerDied","Data":"4154d6792a466183995dd010282e16b08093f1bc3de169a8e4c876b98f77ef46"}
Jul 22 02:36:43.918588 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.918357    2583 scope.go:117] "RemoveContainer" containerID="48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26"
Jul 22 02:36:43.927156 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.927140    2583 scope.go:117] "RemoveContainer" containerID="48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26"
Jul 22 02:36:43.927421 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:36:43.927403    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26\": container with ID starting with 48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26 not found: ID does not exist" containerID="48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26"
Jul 22 02:36:43.927467 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.927432    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26"} err="failed to get container status \"48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26\": rpc error: code = NotFound desc = could not find container \"48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26\": container with ID starting with 48edae61adaca73b71a88838d77db8b6a8071a94abfa5c0fcd7ce1c4cb758d26 not found: ID does not exist"
Jul 22 02:36:43.939413 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.939389    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2482"]
Jul 22 02:36:43.941586 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:43.941568    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-x2482"]
Jul 22 02:36:44.873132 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:44.873090    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0515ea8b-63c5-462e-b04e-1a467c5e86d4" path="/var/lib/kubelet/pods/0515ea8b-63c5-462e-b04e-1a467c5e86d4/volumes"
Jul 22 02:36:47.263668 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.263636    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"]
Jul 22 02:36:47.264079 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.263941    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0515ea8b-63c5-462e-b04e-1a467c5e86d4" containerName="limitador"
Jul 22 02:36:47.264079 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.263952    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0515ea8b-63c5-462e-b04e-1a467c5e86d4" containerName="limitador"
Jul 22 02:36:47.264079 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.263963    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c44f7f7-59f9-4cdc-8081-b04a63a82f89" containerName="authorino"
Jul 22 02:36:47.264079 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.263968    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c44f7f7-59f9-4cdc-8081-b04a63a82f89" containerName="authorino"
Jul 22 02:36:47.264079 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.264052    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c44f7f7-59f9-4cdc-8081-b04a63a82f89" containerName="authorino"
Jul 22 02:36:47.264079 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.264059    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0515ea8b-63c5-462e-b04e-1a467c5e86d4" containerName="limitador"
Jul 22 02:36:47.268090 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.268067    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:47.270724 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.270700    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 22 02:36:47.271528 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.271510    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 22 02:36:47.271626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.271534    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5zkf8\""
Jul 22 02:36:47.271626 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.271534    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 22 02:36:47.273013 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.272990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/30019bad-195a-4e7b-8406-45f3772c4e20-data\") pod \"postgres-69fb4594cf-nkhww\" (UID: \"30019bad-195a-4e7b-8406-45f3772c4e20\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:47.273152 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.273077    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwbdj\" (UniqueName: \"kubernetes.io/projected/30019bad-195a-4e7b-8406-45f3772c4e20-kube-api-access-rwbdj\") pod \"postgres-69fb4594cf-nkhww\" (UID: \"30019bad-195a-4e7b-8406-45f3772c4e20\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:47.274902 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.274844    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"]
Jul 22 02:36:47.373663 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.373627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/30019bad-195a-4e7b-8406-45f3772c4e20-data\") pod \"postgres-69fb4594cf-nkhww\" (UID: \"30019bad-195a-4e7b-8406-45f3772c4e20\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:47.373855 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.373697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbdj\" (UniqueName: \"kubernetes.io/projected/30019bad-195a-4e7b-8406-45f3772c4e20-kube-api-access-rwbdj\") pod \"postgres-69fb4594cf-nkhww\" (UID: \"30019bad-195a-4e7b-8406-45f3772c4e20\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:47.374143 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.374119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/30019bad-195a-4e7b-8406-45f3772c4e20-data\") pod \"postgres-69fb4594cf-nkhww\" (UID: \"30019bad-195a-4e7b-8406-45f3772c4e20\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:47.381561 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.381539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbdj\" (UniqueName: \"kubernetes.io/projected/30019bad-195a-4e7b-8406-45f3772c4e20-kube-api-access-rwbdj\") pod \"postgres-69fb4594cf-nkhww\" (UID: \"30019bad-195a-4e7b-8406-45f3772c4e20\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:47.580159 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.580066    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:47.706549 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.706525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"]
Jul 22 02:36:47.709193 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:36:47.709157    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30019bad_195a_4e7b_8406_45f3772c4e20.slice/crio-d5a4cd1edd52f858a4ca1dcb57a988a4b0234d2aae485df8fb22885a30ebac48 WatchSource:0}: Error finding container d5a4cd1edd52f858a4ca1dcb57a988a4b0234d2aae485df8fb22885a30ebac48: Status 404 returned error can't find the container with id d5a4cd1edd52f858a4ca1dcb57a988a4b0234d2aae485df8fb22885a30ebac48
Jul 22 02:36:47.938488 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:47.938403    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww" event={"ID":"30019bad-195a-4e7b-8406-45f3772c4e20","Type":"ContainerStarted","Data":"d5a4cd1edd52f858a4ca1dcb57a988a4b0234d2aae485df8fb22885a30ebac48"}
Jul 22 02:36:52.965858 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:52.965773    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww" event={"ID":"30019bad-195a-4e7b-8406-45f3772c4e20","Type":"ContainerStarted","Data":"f127a6fb34073dfd4f6023ab4b4804ac0b95bb3523d8c517f97f8a85ce33ee01"}
Jul 22 02:36:52.966216 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:52.965897    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:52.981697 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:52.981648    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww" podStartSLOduration=1.015241505 podStartE2EDuration="5.981632802s" totalImagesPullingTime="4.966391297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:36:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:36:52.981233236 +0000 UTC m=+502.661966281" watchObservedRunningTime="2026-07-22 02:36:52.981632802 +0000 UTC m=+502.662365847"
Jul 22 02:36:59.000181 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.000152    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nkhww"
Jul 22 02:36:59.575576 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.575541    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6z8z2"]
Jul 22 02:36:59.581317 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.581300    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6z8z2"
Jul 22 02:36:59.583961 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.583936    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gfzxx\""
Jul 22 02:36:59.585880 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.585856    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6z8z2"]
Jul 22 02:36:59.664395 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.664359    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7xx\" (UniqueName: \"kubernetes.io/projected/6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0-kube-api-access-rk7xx\") pod \"authorino-5d4b67c446-6z8z2\" (UID: \"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0\") " pod="kuadrant-system/authorino-5d4b67c446-6z8z2"
Jul 22 02:36:59.765731 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.765687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7xx\" (UniqueName: \"kubernetes.io/projected/6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0-kube-api-access-rk7xx\") pod \"authorino-5d4b67c446-6z8z2\" (UID: \"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0\") " pod="kuadrant-system/authorino-5d4b67c446-6z8z2"
Jul 22 02:36:59.776167 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.776134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7xx\" (UniqueName: \"kubernetes.io/projected/6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0-kube-api-access-rk7xx\") pod \"authorino-5d4b67c446-6z8z2\" (UID: \"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0\") " pod="kuadrant-system/authorino-5d4b67c446-6z8z2"
Jul 22 02:36:59.823168 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.823132    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6z8z2"]
Jul 22 02:36:59.823420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.823404    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6z8z2"
Jul 22 02:36:59.852206 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.852176    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-89489d97f-s6hdt"]
Jul 22 02:36:59.856970 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.856953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:36:59.859663 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.859642    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 22 02:36:59.864848 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.864743    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-89489d97f-s6hdt"]
Jul 22 02:36:59.959216 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.959192    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6z8z2"]
Jul 22 02:36:59.961697 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:36:59.961663    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a48b7b9_d2ac_4ab4_8bd9_f281cc8effc0.slice/crio-30616b8af6a93a7b9b809a44216a763b0dedb5c1f0ca491937e50a116362686a WatchSource:0}: Error finding container 30616b8af6a93a7b9b809a44216a763b0dedb5c1f0ca491937e50a116362686a: Status 404 returned error can't find the container with id 30616b8af6a93a7b9b809a44216a763b0dedb5c1f0ca491937e50a116362686a
Jul 22 02:36:59.967372 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.967346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4zfg\" (UniqueName: \"kubernetes.io/projected/3eaef4de-adef-4cfe-86da-84029e304533-kube-api-access-d4zfg\") pod \"authorino-89489d97f-s6hdt\" (UID: \"3eaef4de-adef-4cfe-86da-84029e304533\") " pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:36:59.967475 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.967438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3eaef4de-adef-4cfe-86da-84029e304533-tls-cert\") pod \"authorino-89489d97f-s6hdt\" (UID: \"3eaef4de-adef-4cfe-86da-84029e304533\") " pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:36:59.997047 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:36:59.997003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6z8z2" event={"ID":"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0","Type":"ContainerStarted","Data":"30616b8af6a93a7b9b809a44216a763b0dedb5c1f0ca491937e50a116362686a"}
Jul 22 02:37:00.068877 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:00.068846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4zfg\" (UniqueName: \"kubernetes.io/projected/3eaef4de-adef-4cfe-86da-84029e304533-kube-api-access-d4zfg\") pod \"authorino-89489d97f-s6hdt\" (UID: \"3eaef4de-adef-4cfe-86da-84029e304533\") " pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:37:00.069364 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:00.068897    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3eaef4de-adef-4cfe-86da-84029e304533-tls-cert\") pod \"authorino-89489d97f-s6hdt\" (UID: \"3eaef4de-adef-4cfe-86da-84029e304533\") " pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:37:00.071657 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:00.071635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3eaef4de-adef-4cfe-86da-84029e304533-tls-cert\") pod \"authorino-89489d97f-s6hdt\" (UID: \"3eaef4de-adef-4cfe-86da-84029e304533\") " pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:37:00.078309 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:00.078250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4zfg\" (UniqueName: \"kubernetes.io/projected/3eaef4de-adef-4cfe-86da-84029e304533-kube-api-access-d4zfg\") pod \"authorino-89489d97f-s6hdt\" (UID: \"3eaef4de-adef-4cfe-86da-84029e304533\") " pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:37:00.167796 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:00.167757    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:37:00.296579 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:00.296553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-89489d97f-s6hdt"]
Jul 22 02:37:00.299830 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:37:00.299802    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eaef4de_adef_4cfe_86da_84029e304533.slice/crio-2d6c3fbb60318ac70a8d9a88a7593f05c87e0da5f2a4afa705969cc04c774fc5 WatchSource:0}: Error finding container 2d6c3fbb60318ac70a8d9a88a7593f05c87e0da5f2a4afa705969cc04c774fc5: Status 404 returned error can't find the container with id 2d6c3fbb60318ac70a8d9a88a7593f05c87e0da5f2a4afa705969cc04c774fc5
Jul 22 02:37:01.002610 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.002573    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-89489d97f-s6hdt" event={"ID":"3eaef4de-adef-4cfe-86da-84029e304533","Type":"ContainerStarted","Data":"9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7"}
Jul 22 02:37:01.002805 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.002620    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-89489d97f-s6hdt" event={"ID":"3eaef4de-adef-4cfe-86da-84029e304533","Type":"ContainerStarted","Data":"2d6c3fbb60318ac70a8d9a88a7593f05c87e0da5f2a4afa705969cc04c774fc5"}
Jul 22 02:37:01.004131 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.004104    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6z8z2" event={"ID":"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0","Type":"ContainerStarted","Data":"e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050"}
Jul 22 02:37:01.004268 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.004251    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-6z8z2" podUID="6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0" containerName="authorino" containerID="cri-o://e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050" gracePeriod=30
Jul 22 02:37:01.019542 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.019496    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-89489d97f-s6hdt" podStartSLOduration=1.661465179 podStartE2EDuration="2.019480793s" totalImagesPullingTime="358.015614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:01.018510242 +0000 UTC m=+510.699243290" watchObservedRunningTime="2026-07-22 02:37:01.019480793 +0000 UTC m=+510.700213838"
Jul 22 02:37:01.033656 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.033616    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-6z8z2" podStartSLOduration=1.712801257 podStartE2EDuration="2.033600201s" totalImagesPullingTime="320.798944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:01.033450509 +0000 UTC m=+510.714183556" watchObservedRunningTime="2026-07-22 02:37:01.033600201 +0000 UTC m=+510.714333247"
Jul 22 02:37:01.244866 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.244837    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6z8z2"
Jul 22 02:37:01.280765 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.280689    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk7xx\" (UniqueName: \"kubernetes.io/projected/6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0-kube-api-access-rk7xx\") pod \"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0\" (UID: \"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0\") "
Jul 22 02:37:01.282806 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.282784    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0-kube-api-access-rk7xx" (OuterVolumeSpecName: "kube-api-access-rk7xx") pod "6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0" (UID: "6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0"). InnerVolumeSpecName "kube-api-access-rk7xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:37:01.382112 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:01.382072    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk7xx\" (UniqueName: \"kubernetes.io/projected/6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0-kube-api-access-rk7xx\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:37:02.009735 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.009701    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0" containerID="e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050" exitCode=0
Jul 22 02:37:02.009905 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.009746    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6z8z2"
Jul 22 02:37:02.009905 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.009783    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6z8z2" event={"ID":"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0","Type":"ContainerDied","Data":"e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050"}
Jul 22 02:37:02.009905 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.009823    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-6z8z2" event={"ID":"6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0","Type":"ContainerDied","Data":"30616b8af6a93a7b9b809a44216a763b0dedb5c1f0ca491937e50a116362686a"}
Jul 22 02:37:02.009905 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.009840    2583 scope.go:117] "RemoveContainer" containerID="e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050"
Jul 22 02:37:02.018368 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.018349    2583 scope.go:117] "RemoveContainer" containerID="e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050"
Jul 22 02:37:02.018632 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:37:02.018611    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050\": container with ID starting with e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050 not found: ID does not exist" containerID="e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050"
Jul 22 02:37:02.018685 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.018643    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050"} err="failed to get container status \"e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050\": rpc error: code = NotFound desc = could not find container \"e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050\": container with ID starting with e9ef68548d66bb66a02af15927a3e1fbb7403f4d5c2611d95deca6dc3538d050 not found: ID does not exist"
Jul 22 02:37:02.030528 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.030505    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6z8z2"]
Jul 22 02:37:02.035043 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.035006    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6z8z2"]
Jul 22 02:37:02.868166 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:02.868133    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0" path="/var/lib/kubelet/pods/6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0/volumes"
Jul 22 02:37:43.502505 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.502472    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"]
Jul 22 02:37:43.502903 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.502781    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0" containerName="authorino"
Jul 22 02:37:43.502903 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.502791    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0" containerName="authorino"
Jul 22 02:37:43.502903 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.502852    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a48b7b9-d2ac-4ab4-8bd9-f281cc8effc0" containerName="authorino"
Jul 22 02:37:43.506970 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.506954    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.510570 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.510547    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 22 02:37:43.510663 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.510551    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g58cb\""
Jul 22 02:37:43.516097 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.516070    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"]
Jul 22 02:37:43.534411 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.534372    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab2a44f-66ee-424c-a923-1ad334f56918-openshift-service-ca\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.534575 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.534438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ab2a44f-66ee-424c-a923-1ad334f56918-maas-api-tls\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.534575 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.534515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb6mk\" (UniqueName: \"kubernetes.io/projected/3ab2a44f-66ee-424c-a923-1ad334f56918-kube-api-access-lb6mk\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.635382 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.635349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ab2a44f-66ee-424c-a923-1ad334f56918-maas-api-tls\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.635542 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.635387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb6mk\" (UniqueName: \"kubernetes.io/projected/3ab2a44f-66ee-424c-a923-1ad334f56918-kube-api-access-lb6mk\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.635542 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.635445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab2a44f-66ee-424c-a923-1ad334f56918-openshift-service-ca\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.636139 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.636121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ab2a44f-66ee-424c-a923-1ad334f56918-openshift-service-ca\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.637919 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.637895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ab2a44f-66ee-424c-a923-1ad334f56918-maas-api-tls\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.644628 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.644604    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb6mk\" (UniqueName: \"kubernetes.io/projected/3ab2a44f-66ee-424c-a923-1ad334f56918-kube-api-access-lb6mk\") pod \"maas-api-7c4dcdb54c-k42cz\" (UID: \"3ab2a44f-66ee-424c-a923-1ad334f56918\") " pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.818423 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.818334    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:43.958540 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.958508    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"]
Jul 22 02:37:43.961688 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:37:43.961660    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab2a44f_66ee_424c_a923_1ad334f56918.slice/crio-f98729e6d4fb1f8c5e45e9b654419629181a108384c03887d442fab4eb0485fb WatchSource:0}: Error finding container f98729e6d4fb1f8c5e45e9b654419629181a108384c03887d442fab4eb0485fb: Status 404 returned error can't find the container with id f98729e6d4fb1f8c5e45e9b654419629181a108384c03887d442fab4eb0485fb
Jul 22 02:37:43.994983 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.994946    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-9fdns"]
Jul 22 02:37:43.999371 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:43.999347    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:44.002117 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.002092    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 22 02:37:44.002530 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.002361    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9q4jf\""
Jul 22 02:37:44.002530 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.002480    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-9fdns"]
Jul 22 02:37:44.038965 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.038928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82bb7506-cf1e-4e7c-b747-abc86314f7ad-plugins-config-volume\") pod \"payload-processing-5bf47cd988-9fdns\" (UID: \"82bb7506-cf1e-4e7c-b747-abc86314f7ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:44.039131 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.038975    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6tc\" (UniqueName: \"kubernetes.io/projected/82bb7506-cf1e-4e7c-b747-abc86314f7ad-kube-api-access-bf6tc\") pod \"payload-processing-5bf47cd988-9fdns\" (UID: \"82bb7506-cf1e-4e7c-b747-abc86314f7ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:44.140914 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.140819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82bb7506-cf1e-4e7c-b747-abc86314f7ad-plugins-config-volume\") pod \"payload-processing-5bf47cd988-9fdns\" (UID: \"82bb7506-cf1e-4e7c-b747-abc86314f7ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:44.140914 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.140866    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6tc\" (UniqueName: \"kubernetes.io/projected/82bb7506-cf1e-4e7c-b747-abc86314f7ad-kube-api-access-bf6tc\") pod \"payload-processing-5bf47cd988-9fdns\" (UID: \"82bb7506-cf1e-4e7c-b747-abc86314f7ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:44.141699 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.141677    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82bb7506-cf1e-4e7c-b747-abc86314f7ad-plugins-config-volume\") pod \"payload-processing-5bf47cd988-9fdns\" (UID: \"82bb7506-cf1e-4e7c-b747-abc86314f7ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:44.150721 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.150697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6tc\" (UniqueName: \"kubernetes.io/projected/82bb7506-cf1e-4e7c-b747-abc86314f7ad-kube-api-access-bf6tc\") pod \"payload-processing-5bf47cd988-9fdns\" (UID: \"82bb7506-cf1e-4e7c-b747-abc86314f7ad\") " pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:44.184884 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.184848    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz" event={"ID":"3ab2a44f-66ee-424c-a923-1ad334f56918","Type":"ContainerStarted","Data":"f98729e6d4fb1f8c5e45e9b654419629181a108384c03887d442fab4eb0485fb"}
Jul 22 02:37:44.312134 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.312094    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:44.442672 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:44.442512    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-9fdns"]
Jul 22 02:37:44.445231 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:37:44.445195    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bb7506_cf1e_4e7c_b747_abc86314f7ad.slice/crio-1224e8f73a0cb5a501aecdb80c79b14dfef2ee6cccf76a59bede5ec9b6a421b2 WatchSource:0}: Error finding container 1224e8f73a0cb5a501aecdb80c79b14dfef2ee6cccf76a59bede5ec9b6a421b2: Status 404 returned error can't find the container with id 1224e8f73a0cb5a501aecdb80c79b14dfef2ee6cccf76a59bede5ec9b6a421b2
Jul 22 02:37:45.192746 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:45.192703    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-9fdns" event={"ID":"82bb7506-cf1e-4e7c-b747-abc86314f7ad","Type":"ContainerStarted","Data":"1224e8f73a0cb5a501aecdb80c79b14dfef2ee6cccf76a59bede5ec9b6a421b2"}
Jul 22 02:37:47.202711 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:47.202674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz" event={"ID":"3ab2a44f-66ee-424c-a923-1ad334f56918","Type":"ContainerStarted","Data":"9acdf17ab3687800ef75668a21416197d3105585182044583a69d371897532dc"}
Jul 22 02:37:47.203210 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:47.202757    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:47.204193 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:47.204174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-9fdns" event={"ID":"82bb7506-cf1e-4e7c-b747-abc86314f7ad","Type":"ContainerStarted","Data":"8b9bef003f83f3b1b9e8d6d75d933d2d4c4afab9cd649241a4fe73e1fe1e2cff"}
Jul 22 02:37:47.204290 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:47.204211    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:37:47.221120 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:47.221067    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz" podStartSLOduration=1.499563291 podStartE2EDuration="4.221052615s" totalImagesPullingTime="2.721489324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:47.21984259 +0000 UTC m=+556.900575635" watchObservedRunningTime="2026-07-22 02:37:47.221052615 +0000 UTC m=+556.901785656"
Jul 22 02:37:47.236453 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:47.236393    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-9fdns" podStartSLOduration=1.997506033 podStartE2EDuration="4.236370086s" totalImagesPullingTime="2.238864053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:37:47.235920221 +0000 UTC m=+556.916653266" watchObservedRunningTime="2026-07-22 02:37:47.236370086 +0000 UTC m=+556.917103131"
Jul 22 02:37:53.213985 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:53.213953    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7c4dcdb54c-k42cz"
Jul 22 02:37:58.210656 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:37:58.210627    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-9fdns"
Jul 22 02:38:30.726582 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:38:30.726544    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:38:30.727083 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:38:30.726748    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:39:11.905005 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:11.904926    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"]
Jul 22 02:39:11.908414 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:11.908399    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:11.910831 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:11.910810    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pj5rv\""
Jul 22 02:39:11.912114 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:11.912091    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 22 02:39:11.912917 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:11.912897    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 22 02:39:11.913596 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:11.913580    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 22 02:39:11.923635 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:11.923609    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"]
Jul 22 02:39:12.005693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.005654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.005693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.005696    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.005912 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.005767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvz2v\" (UniqueName: \"kubernetes.io/projected/20f83412-e6e4-4341-8dd5-666c863548bb-kube-api-access-kvz2v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.005912 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.005798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.005912 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.005850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.005912 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.005874    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f83412-e6e4-4341-8dd5-666c863548bb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107163 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvz2v\" (UniqueName: \"kubernetes.io/projected/20f83412-e6e4-4341-8dd5-666c863548bb-kube-api-access-kvz2v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107329 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107329 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107329 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f83412-e6e4-4341-8dd5-666c863548bb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107486 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107486 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107643 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107831 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.107961 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.107854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.109921 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.109893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20f83412-e6e4-4341-8dd5-666c863548bb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.110131 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.110108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20f83412-e6e4-4341-8dd5-666c863548bb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.128475 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.128450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvz2v\" (UniqueName: \"kubernetes.io/projected/20f83412-e6e4-4341-8dd5-666c863548bb-kube-api-access-kvz2v\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zh4vf\" (UID: \"20f83412-e6e4-4341-8dd5-666c863548bb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.218808 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.218704    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:12.360119 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.360089    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"]
Jul 22 02:39:12.362984 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:39:12.362960    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f83412_e6e4_4341_8dd5_666c863548bb.slice/crio-45250d209c0d4de49fb746ac41085748b3cc7ab65447072d33044553136235cc WatchSource:0}: Error finding container 45250d209c0d4de49fb746ac41085748b3cc7ab65447072d33044553136235cc: Status 404 returned error can't find the container with id 45250d209c0d4de49fb746ac41085748b3cc7ab65447072d33044553136235cc
Jul 22 02:39:12.364198 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.364182    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:39:12.581739 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:12.581686    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf" event={"ID":"20f83412-e6e4-4341-8dd5-666c863548bb","Type":"ContainerStarted","Data":"45250d209c0d4de49fb746ac41085748b3cc7ab65447072d33044553136235cc"}
Jul 22 02:39:13.587262 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:13.587228    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf" event={"ID":"20f83412-e6e4-4341-8dd5-666c863548bb","Type":"ContainerStarted","Data":"42b74999d7f074484ea5f958e6ea338e2263c4ef74cb0ea6ff95cd4614ed8dfc"}
Jul 22 02:39:13.613582 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:13.613516    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf" podStartSLOduration=1.585916261 podStartE2EDuration="2.613498944s" totalImagesPullingTime="1.027582683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:39:13.611880464 +0000 UTC m=+643.292613511" watchObservedRunningTime="2026-07-22 02:39:13.613498944 +0000 UTC m=+643.294232016"
Jul 22 02:39:19.386078 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.386046    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"]
Jul 22 02:39:19.389414 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.389390    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.391896 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.391873    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 22 02:39:19.401371 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.401348    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"]
Jul 22 02:39:19.577638 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.577603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.577638 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.577635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.577900 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.577667    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d56769-7809-46f2-bbad-5cbb65e85fee-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.577900 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.577779    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.577900 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.577817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff6vp\" (UniqueName: \"kubernetes.io/projected/90d56769-7809-46f2-bbad-5cbb65e85fee-kube-api-access-ff6vp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.577900 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.577860    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.678887 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.678784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.678887 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.678825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff6vp\" (UniqueName: \"kubernetes.io/projected/90d56769-7809-46f2-bbad-5cbb65e85fee-kube-api-access-ff6vp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.678887 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.678851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.679225 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.678911    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.679225 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.678928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.679225 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.678965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d56769-7809-46f2-bbad-5cbb65e85fee-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.679373 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.679324    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.679373 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.679358    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.679443 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.679391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.681384 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.681357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d56769-7809-46f2-bbad-5cbb65e85fee-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.681747 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.681725    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d56769-7809-46f2-bbad-5cbb65e85fee-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.686972 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.686947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff6vp\" (UniqueName: \"kubernetes.io/projected/90d56769-7809-46f2-bbad-5cbb65e85fee-kube-api-access-ff6vp\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq\" (UID: \"90d56769-7809-46f2-bbad-5cbb65e85fee\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.698864 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.698841    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:19.833330 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:19.833270    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"]
Jul 22 02:39:19.836115 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:39:19.836079    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d56769_7809_46f2_bbad_5cbb65e85fee.slice/crio-024d98c883a6de5b0c14262d358e8cd60d5218562193fc56ffe4aafb740dad28 WatchSource:0}: Error finding container 024d98c883a6de5b0c14262d358e8cd60d5218562193fc56ffe4aafb740dad28: Status 404 returned error can't find the container with id 024d98c883a6de5b0c14262d358e8cd60d5218562193fc56ffe4aafb740dad28
Jul 22 02:39:20.623460 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:20.623427    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq" event={"ID":"90d56769-7809-46f2-bbad-5cbb65e85fee","Type":"ContainerStarted","Data":"e2ad065e6be669683ec2276b1c97eaabba60c674a01303cddb2291b868e5161f"}
Jul 22 02:39:20.623460 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:20.623468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq" event={"ID":"90d56769-7809-46f2-bbad-5cbb65e85fee","Type":"ContainerStarted","Data":"024d98c883a6de5b0c14262d358e8cd60d5218562193fc56ffe4aafb740dad28"}
Jul 22 02:39:20.650399 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:20.650345    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq" podStartSLOduration=1.452516278 podStartE2EDuration="1.650329114s" totalImagesPullingTime="197.812836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:39:20.648403029 +0000 UTC m=+650.329136074" watchObservedRunningTime="2026-07-22 02:39:20.650329114 +0000 UTC m=+650.331062160"
Jul 22 02:39:22.219328 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:22.219283    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:22.219691 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:22.219348    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:22.232180 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:22.232152    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:22.645261 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:22.645229    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zh4vf"
Jul 22 02:39:29.699692 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:29.699657    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:29.699692 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:29.699698    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:29.712470 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:29.712446    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:39:30.678429 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:39:30.678402    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq"
Jul 22 02:40:42.384327 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.384291    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67d8987c54-zl6mw"]
Jul 22 02:40:42.387752 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.387735    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67d8987c54-zl6mw"
Jul 22 02:40:42.396241 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.396212    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67d8987c54-zl6mw"]
Jul 22 02:40:42.420254 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.420212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bkq\" (UniqueName: \"kubernetes.io/projected/5ef93db4-29c6-454c-8641-83f0d1fef925-kube-api-access-67bkq\") pod \"authorino-67d8987c54-zl6mw\" (UID: \"5ef93db4-29c6-454c-8641-83f0d1fef925\") " pod="kuadrant-system/authorino-67d8987c54-zl6mw"
Jul 22 02:40:42.420414 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.420269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5ef93db4-29c6-454c-8641-83f0d1fef925-tls-cert\") pod \"authorino-67d8987c54-zl6mw\" (UID: \"5ef93db4-29c6-454c-8641-83f0d1fef925\") " pod="kuadrant-system/authorino-67d8987c54-zl6mw"
Jul 22 02:40:42.521254 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.521217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5ef93db4-29c6-454c-8641-83f0d1fef925-tls-cert\") pod \"authorino-67d8987c54-zl6mw\" (UID: \"5ef93db4-29c6-454c-8641-83f0d1fef925\") " pod="kuadrant-system/authorino-67d8987c54-zl6mw"
Jul 22 02:40:42.521440 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.521320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bkq\" (UniqueName: \"kubernetes.io/projected/5ef93db4-29c6-454c-8641-83f0d1fef925-kube-api-access-67bkq\") pod \"authorino-67d8987c54-zl6mw\" (UID: \"5ef93db4-29c6-454c-8641-83f0d1fef925\") " pod="kuadrant-system/authorino-67d8987c54-zl6mw"
Jul 22 02:40:42.523796 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.523774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5ef93db4-29c6-454c-8641-83f0d1fef925-tls-cert\") pod \"authorino-67d8987c54-zl6mw\" (UID: \"5ef93db4-29c6-454c-8641-83f0d1fef925\") " pod="kuadrant-system/authorino-67d8987c54-zl6mw"
Jul 22 02:40:42.533557 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.533530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bkq\" (UniqueName: \"kubernetes.io/projected/5ef93db4-29c6-454c-8641-83f0d1fef925-kube-api-access-67bkq\") pod \"authorino-67d8987c54-zl6mw\" (UID: \"5ef93db4-29c6-454c-8641-83f0d1fef925\") " pod="kuadrant-system/authorino-67d8987c54-zl6mw"
Jul 22 02:40:42.697917 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.697832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67d8987c54-zl6mw"
Jul 22 02:40:42.847354 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.847331    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67d8987c54-zl6mw"]
Jul 22 02:40:42.849890 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:40:42.849857    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ef93db4_29c6_454c_8641_83f0d1fef925.slice/crio-06cb429621c29207d8b836540f86e4b474e9db174f060756fe6118d5886bc196 WatchSource:0}: Error finding container 06cb429621c29207d8b836540f86e4b474e9db174f060756fe6118d5886bc196: Status 404 returned error can't find the container with id 06cb429621c29207d8b836540f86e4b474e9db174f060756fe6118d5886bc196
Jul 22 02:40:42.988499 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:42.988456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67d8987c54-zl6mw" event={"ID":"5ef93db4-29c6-454c-8641-83f0d1fef925","Type":"ContainerStarted","Data":"06cb429621c29207d8b836540f86e4b474e9db174f060756fe6118d5886bc196"}
Jul 22 02:40:43.994713 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:43.994680    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67d8987c54-zl6mw" event={"ID":"5ef93db4-29c6-454c-8641-83f0d1fef925","Type":"ContainerStarted","Data":"3c3c18d9a0d9ddab3cab2a7afaf284eb1fee2a2e9de141d64db2a9eca2b4196d"}
Jul 22 02:40:44.025243 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.025189    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-67d8987c54-zl6mw" podStartSLOduration=1.5174153929999998 podStartE2EDuration="2.025168098s" totalImagesPullingTime="507.752705ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:40:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:40:44.023653031 +0000 UTC m=+733.704386080" watchObservedRunningTime="2026-07-22 02:40:44.025168098 +0000 UTC m=+733.705901121"
Jul 22 02:40:44.078246 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.078202    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-89489d97f-s6hdt"]
Jul 22 02:40:44.079190 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.078425    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-89489d97f-s6hdt" podUID="3eaef4de-adef-4cfe-86da-84029e304533" containerName="authorino" containerID="cri-o://9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7" gracePeriod=30
Jul 22 02:40:44.352799 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.352769    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:40:44.439228 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.439189    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3eaef4de-adef-4cfe-86da-84029e304533-tls-cert\") pod \"3eaef4de-adef-4cfe-86da-84029e304533\" (UID: \"3eaef4de-adef-4cfe-86da-84029e304533\") "
Jul 22 02:40:44.439228 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.439229    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4zfg\" (UniqueName: \"kubernetes.io/projected/3eaef4de-adef-4cfe-86da-84029e304533-kube-api-access-d4zfg\") pod \"3eaef4de-adef-4cfe-86da-84029e304533\" (UID: \"3eaef4de-adef-4cfe-86da-84029e304533\") "
Jul 22 02:40:44.441540 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.441504    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eaef4de-adef-4cfe-86da-84029e304533-kube-api-access-d4zfg" (OuterVolumeSpecName: "kube-api-access-d4zfg") pod "3eaef4de-adef-4cfe-86da-84029e304533" (UID: "3eaef4de-adef-4cfe-86da-84029e304533"). InnerVolumeSpecName "kube-api-access-d4zfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:40:44.450515 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.450489    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eaef4de-adef-4cfe-86da-84029e304533-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3eaef4de-adef-4cfe-86da-84029e304533" (UID: "3eaef4de-adef-4cfe-86da-84029e304533"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:40:44.540449 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.540412    2583 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3eaef4de-adef-4cfe-86da-84029e304533-tls-cert\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:40:44.540449 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:44.540438    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4zfg\" (UniqueName: \"kubernetes.io/projected/3eaef4de-adef-4cfe-86da-84029e304533-kube-api-access-d4zfg\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:40:45.002385 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.002348    2583 generic.go:356] "Generic (PLEG): container finished" podID="3eaef4de-adef-4cfe-86da-84029e304533" containerID="9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7" exitCode=0
Jul 22 02:40:45.002789 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.002392    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-89489d97f-s6hdt"
Jul 22 02:40:45.002789 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.002436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-89489d97f-s6hdt" event={"ID":"3eaef4de-adef-4cfe-86da-84029e304533","Type":"ContainerDied","Data":"9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7"}
Jul 22 02:40:45.002789 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.002474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-89489d97f-s6hdt" event={"ID":"3eaef4de-adef-4cfe-86da-84029e304533","Type":"ContainerDied","Data":"2d6c3fbb60318ac70a8d9a88a7593f05c87e0da5f2a4afa705969cc04c774fc5"}
Jul 22 02:40:45.002789 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.002495    2583 scope.go:117] "RemoveContainer" containerID="9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7"
Jul 22 02:40:45.011177 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.011147    2583 scope.go:117] "RemoveContainer" containerID="9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7"
Jul 22 02:40:45.011539 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:40:45.011520    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7\": container with ID starting with 9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7 not found: ID does not exist" containerID="9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7"
Jul 22 02:40:45.011616 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.011554    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7"} err="failed to get container status \"9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7\": rpc error: code = NotFound desc = could not find container \"9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7\": container with ID starting with 9698a84222526cc7154ac6b41a3856ead4896d1e476c305ea183866812b99ee7 not found: ID does not exist"
Jul 22 02:40:45.020299 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.020267    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-89489d97f-s6hdt"]
Jul 22 02:40:45.022651 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:45.022628    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-89489d97f-s6hdt"]
Jul 22 02:40:46.873002 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:40:46.872966    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eaef4de-adef-4cfe-86da-84029e304533" path="/var/lib/kubelet/pods/3eaef4de-adef-4cfe-86da-84029e304533/volumes"
Jul 22 02:43:30.758871 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:43:30.758841    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:43:30.760734 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:43:30.760712    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:48:30.784675 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:48:30.784641    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:48:30.787657 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:48:30.787633    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:51:19.031561 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.031482    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"]
Jul 22 02:51:19.033989 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.031816    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eaef4de-adef-4cfe-86da-84029e304533" containerName="authorino"
Jul 22 02:51:19.033989 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.031827    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eaef4de-adef-4cfe-86da-84029e304533" containerName="authorino"
Jul 22 02:51:19.033989 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.031878    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eaef4de-adef-4cfe-86da-84029e304533" containerName="authorino"
Jul 22 02:51:19.035018 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.035003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.039234 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.039215    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d9228d76-kserve-self-signed-certs\""
Jul 22 02:51:19.063403 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.063374    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"]
Jul 22 02:51:19.145312 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.145276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-dshm\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.145312 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.145314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-tmp-dir\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.145544 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.145340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvwb\" (UniqueName: \"kubernetes.io/projected/216a5b3a-0ce8-43e0-b408-329751e775c8-kube-api-access-chvwb\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.145544 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.145436    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-home\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.145544 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.145528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216a5b3a-0ce8-43e0-b408-329751e775c8-tls-certs\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.145652 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.145557    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-model-cache\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.246912 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.246876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-model-cache\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.247172 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.246921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-dshm\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.247172 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.246942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-tmp-dir\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.247172 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.246963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvwb\" (UniqueName: \"kubernetes.io/projected/216a5b3a-0ce8-43e0-b408-329751e775c8-kube-api-access-chvwb\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.247172 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.247130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-home\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.247403 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.247271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216a5b3a-0ce8-43e0-b408-329751e775c8-tls-certs\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.247403 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.247347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-model-cache\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.247505 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.247413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-tmp-dir\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.247629 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.247611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-home\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.249498 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.249473    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-dshm\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.249916 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.249896    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216a5b3a-0ce8-43e0-b408-329751e775c8-tls-certs\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.267328 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.267294    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvwb\" (UniqueName: \"kubernetes.io/projected/216a5b3a-0ce8-43e0-b408-329751e775c8-kube-api-access-chvwb\") pod \"e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") " pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.343833 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.343738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:19.516902 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.516722    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"]
Jul 22 02:51:19.519417 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:51:19.519389    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod216a5b3a_0ce8_43e0_b408_329751e775c8.slice/crio-eb262d4f0f0338ad3533db5f82808727b4aac91107c3e10b593f39042a0c8709 WatchSource:0}: Error finding container eb262d4f0f0338ad3533db5f82808727b4aac91107c3e10b593f39042a0c8709: Status 404 returned error can't find the container with id eb262d4f0f0338ad3533db5f82808727b4aac91107c3e10b593f39042a0c8709
Jul 22 02:51:19.798562 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.798518    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862" event={"ID":"216a5b3a-0ce8-43e0-b408-329751e775c8","Type":"ContainerStarted","Data":"9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726"}
Jul 22 02:51:19.798760 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.798574    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862" event={"ID":"216a5b3a-0ce8-43e0-b408-329751e775c8","Type":"ContainerStarted","Data":"eb262d4f0f0338ad3533db5f82808727b4aac91107c3e10b593f39042a0c8709"}
Jul 22 02:51:19.827144 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:19.827079    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862" podStartSLOduration=0.827056769 podStartE2EDuration="827.056769ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:51:19.826077019 +0000 UTC m=+1369.506810069" watchObservedRunningTime="2026-07-22 02:51:19.827056769 +0000 UTC m=+1369.507789816"
Jul 22 02:51:29.344217 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:29.344129    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:29.344641 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:29.344395    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:29.356519 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:29.356488    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:29.857411 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:29.857383    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:51:55.274347 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:51:55.274305    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"]
Jul 22 02:52:10.295909 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.295857    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862" podUID="216a5b3a-0ce8-43e0-b408-329751e775c8" containerName="main" containerID="cri-o://9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726" gracePeriod=45
Jul 22 02:52:10.547894 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.547829    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:52:10.629425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629382    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chvwb\" (UniqueName: \"kubernetes.io/projected/216a5b3a-0ce8-43e0-b408-329751e775c8-kube-api-access-chvwb\") pod \"216a5b3a-0ce8-43e0-b408-329751e775c8\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") "
Jul 22 02:52:10.629425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629423    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-model-cache\") pod \"216a5b3a-0ce8-43e0-b408-329751e775c8\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") "
Jul 22 02:52:10.629693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629447    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216a5b3a-0ce8-43e0-b408-329751e775c8-tls-certs\") pod \"216a5b3a-0ce8-43e0-b408-329751e775c8\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") "
Jul 22 02:52:10.629693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629493    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-tmp-dir\") pod \"216a5b3a-0ce8-43e0-b408-329751e775c8\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") "
Jul 22 02:52:10.629693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-dshm\") pod \"216a5b3a-0ce8-43e0-b408-329751e775c8\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") "
Jul 22 02:52:10.629693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629586    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-home\") pod \"216a5b3a-0ce8-43e0-b408-329751e775c8\" (UID: \"216a5b3a-0ce8-43e0-b408-329751e775c8\") "
Jul 22 02:52:10.629889 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629709    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "216a5b3a-0ce8-43e0-b408-329751e775c8" (UID: "216a5b3a-0ce8-43e0-b408-329751e775c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:52:10.629941 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629918    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-model-cache\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:52:10.629941 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.629930    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-home" (OuterVolumeSpecName: "home") pod "216a5b3a-0ce8-43e0-b408-329751e775c8" (UID: "216a5b3a-0ce8-43e0-b408-329751e775c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:52:10.630090 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.630061    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "216a5b3a-0ce8-43e0-b408-329751e775c8" (UID: "216a5b3a-0ce8-43e0-b408-329751e775c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:52:10.631909 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.631884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216a5b3a-0ce8-43e0-b408-329751e775c8-kube-api-access-chvwb" (OuterVolumeSpecName: "kube-api-access-chvwb") pod "216a5b3a-0ce8-43e0-b408-329751e775c8" (UID: "216a5b3a-0ce8-43e0-b408-329751e775c8"). InnerVolumeSpecName "kube-api-access-chvwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:52:10.632097 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.632077    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216a5b3a-0ce8-43e0-b408-329751e775c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "216a5b3a-0ce8-43e0-b408-329751e775c8" (UID: "216a5b3a-0ce8-43e0-b408-329751e775c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:52:10.632199 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.632174    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-dshm" (OuterVolumeSpecName: "dshm") pod "216a5b3a-0ce8-43e0-b408-329751e775c8" (UID: "216a5b3a-0ce8-43e0-b408-329751e775c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 02:52:10.731016 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.730983    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chvwb\" (UniqueName: \"kubernetes.io/projected/216a5b3a-0ce8-43e0-b408-329751e775c8-kube-api-access-chvwb\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:52:10.731016 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.731013    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/216a5b3a-0ce8-43e0-b408-329751e775c8-tls-certs\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:52:10.731016 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.731024    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-tmp-dir\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:52:10.731268 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.731053    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-dshm\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:52:10.731268 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:10.731060    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/216a5b3a-0ce8-43e0-b408-329751e775c8-home\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:52:11.035067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.035017    2583 generic.go:356] "Generic (PLEG): container finished" podID="216a5b3a-0ce8-43e0-b408-329751e775c8" containerID="9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726" exitCode=0
Jul 22 02:52:11.035067 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.035078    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"
Jul 22 02:52:11.035269 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.035109    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862" event={"ID":"216a5b3a-0ce8-43e0-b408-329751e775c8","Type":"ContainerDied","Data":"9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726"}
Jul 22 02:52:11.035269 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.035151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862" event={"ID":"216a5b3a-0ce8-43e0-b408-329751e775c8","Type":"ContainerDied","Data":"eb262d4f0f0338ad3533db5f82808727b4aac91107c3e10b593f39042a0c8709"}
Jul 22 02:52:11.035269 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.035170    2583 scope.go:117] "RemoveContainer" containerID="9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726"
Jul 22 02:52:11.043705 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.043688    2583 scope.go:117] "RemoveContainer" containerID="9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726"
Jul 22 02:52:11.043955 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:52:11.043938    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726\": container with ID starting with 9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726 not found: ID does not exist" containerID="9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726"
Jul 22 02:52:11.044020 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.043963    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726"} err="failed to get container status \"9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726\": rpc error: code = NotFound desc = could not find container \"9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726\": container with ID starting with 9df9154daf2ecf2a1c5aeb3ce842a352acb7dafe4ea2782c97b74ba20a19e726 not found: ID does not exist"
Jul 22 02:52:11.051436 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.051414    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"]
Jul 22 02:52:11.057693 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:11.057662    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d9228d76-kserve-6c447ddb79-rt862"]
Jul 22 02:52:12.868075 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:52:12.868016    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="216a5b3a-0ce8-43e0-b408-329751e775c8" path="/var/lib/kubelet/pods/216a5b3a-0ce8-43e0-b408-329751e775c8/volumes"
Jul 22 02:53:30.815124 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:53:30.815095    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:53:30.818636 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:53:30.818614    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:55:38.707204 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.707164    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"]
Jul 22 02:55:38.707696 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.707655    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="216a5b3a-0ce8-43e0-b408-329751e775c8" containerName="main"
Jul 22 02:55:38.707696 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.707670    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="216a5b3a-0ce8-43e0-b408-329751e775c8" containerName="main"
Jul 22 02:55:38.707798 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.707786    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="216a5b3a-0ce8-43e0-b408-329751e775c8" containerName="main"
Jul 22 02:55:38.711215 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.711196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.713696 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.713680    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3c6d7b74-serving-cert\""
Jul 22 02:55:38.721128 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.721099    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"]
Jul 22 02:55:38.741785 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.741753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bl2s\" (UniqueName: \"kubernetes.io/projected/e864f7ee-a354-4b50-be1a-cedfe48ed659-kube-api-access-5bl2s\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.741940 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.741790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e864f7ee-a354-4b50-be1a-cedfe48ed659-openshift-service-ca\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.741940 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.741828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e864f7ee-a354-4b50-be1a-cedfe48ed659-maas-api-tls\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.842856 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.842822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bl2s\" (UniqueName: \"kubernetes.io/projected/e864f7ee-a354-4b50-be1a-cedfe48ed659-kube-api-access-5bl2s\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.843068 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.842865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e864f7ee-a354-4b50-be1a-cedfe48ed659-openshift-service-ca\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.843068 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.842903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e864f7ee-a354-4b50-be1a-cedfe48ed659-maas-api-tls\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.843557 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.843521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e864f7ee-a354-4b50-be1a-cedfe48ed659-openshift-service-ca\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.845468 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.845447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e864f7ee-a354-4b50-be1a-cedfe48ed659-maas-api-tls\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:38.852462 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:38.852435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bl2s\" (UniqueName: \"kubernetes.io/projected/e864f7ee-a354-4b50-be1a-cedfe48ed659-kube-api-access-5bl2s\") pod \"maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:39.022157 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:39.022123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:39.172308 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:39.172278    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"]
Jul 22 02:55:39.175530 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:55:39.175500    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode864f7ee_a354_4b50_be1a_cedfe48ed659.slice/crio-a7f9a7bf012000427ccd7d5934dd59280def7c71c58cb6c4904800e1d71b245b WatchSource:0}: Error finding container a7f9a7bf012000427ccd7d5934dd59280def7c71c58cb6c4904800e1d71b245b: Status 404 returned error can't find the container with id a7f9a7bf012000427ccd7d5934dd59280def7c71c58cb6c4904800e1d71b245b
Jul 22 02:55:39.176817 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:39.176800    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 02:55:39.966407 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:39.966313    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc" event={"ID":"e864f7ee-a354-4b50-be1a-cedfe48ed659","Type":"ContainerStarted","Data":"c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a"}
Jul 22 02:55:39.966407 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:39.966377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc" event={"ID":"e864f7ee-a354-4b50-be1a-cedfe48ed659","Type":"ContainerStarted","Data":"a7f9a7bf012000427ccd7d5934dd59280def7c71c58cb6c4904800e1d71b245b"}
Jul 22 02:55:39.966407 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:39.966403    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:55:39.987306 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:39.987255    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc" podStartSLOduration=1.570717438 podStartE2EDuration="1.987236129s" totalImagesPullingTime="416.518691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:55:39.985836166 +0000 UTC m=+1629.666569214" watchObservedRunningTime="2026-07-22 02:55:39.987236129 +0000 UTC m=+1629.667969176"
Jul 22 02:55:45.976514 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:55:45.976489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:56:14.762633 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:14.762601    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"]
Jul 22 02:56:14.763167 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:14.762804    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc" podUID="e864f7ee-a354-4b50-be1a-cedfe48ed659" containerName="maas-api" containerID="cri-o://c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a" gracePeriod=30
Jul 22 02:56:15.017767 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.017700    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:56:15.122750 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.122713    2583 generic.go:356] "Generic (PLEG): container finished" podID="e864f7ee-a354-4b50-be1a-cedfe48ed659" containerID="c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a" exitCode=0
Jul 22 02:56:15.122911 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.122764    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"
Jul 22 02:56:15.122911 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.122791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc" event={"ID":"e864f7ee-a354-4b50-be1a-cedfe48ed659","Type":"ContainerDied","Data":"c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a"}
Jul 22 02:56:15.122911 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.122844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc" event={"ID":"e864f7ee-a354-4b50-be1a-cedfe48ed659","Type":"ContainerDied","Data":"a7f9a7bf012000427ccd7d5934dd59280def7c71c58cb6c4904800e1d71b245b"}
Jul 22 02:56:15.122911 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.122866    2583 scope.go:117] "RemoveContainer" containerID="c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a"
Jul 22 02:56:15.131753 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.131732    2583 scope.go:117] "RemoveContainer" containerID="c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a"
Jul 22 02:56:15.132089 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:56:15.132063    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a\": container with ID starting with c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a not found: ID does not exist" containerID="c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a"
Jul 22 02:56:15.132184 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.132104    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a"} err="failed to get container status \"c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a\": rpc error: code = NotFound desc = could not find container \"c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a\": container with ID starting with c280b1a2ac1a17667c75fa63a9cec7e9c16e34c1fb007aee3c4fd29e67eed07a not found: ID does not exist"
Jul 22 02:56:15.161534 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.161505    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e864f7ee-a354-4b50-be1a-cedfe48ed659-maas-api-tls\") pod \"e864f7ee-a354-4b50-be1a-cedfe48ed659\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") "
Jul 22 02:56:15.161653 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.161603    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e864f7ee-a354-4b50-be1a-cedfe48ed659-openshift-service-ca\") pod \"e864f7ee-a354-4b50-be1a-cedfe48ed659\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") "
Jul 22 02:56:15.161704 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.161652    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bl2s\" (UniqueName: \"kubernetes.io/projected/e864f7ee-a354-4b50-be1a-cedfe48ed659-kube-api-access-5bl2s\") pod \"e864f7ee-a354-4b50-be1a-cedfe48ed659\" (UID: \"e864f7ee-a354-4b50-be1a-cedfe48ed659\") "
Jul 22 02:56:15.161981 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.161960    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e864f7ee-a354-4b50-be1a-cedfe48ed659-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e864f7ee-a354-4b50-be1a-cedfe48ed659" (UID: "e864f7ee-a354-4b50-be1a-cedfe48ed659"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:56:15.163722 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.163696    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e864f7ee-a354-4b50-be1a-cedfe48ed659-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e864f7ee-a354-4b50-be1a-cedfe48ed659" (UID: "e864f7ee-a354-4b50-be1a-cedfe48ed659"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:56:15.163892 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.163870    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e864f7ee-a354-4b50-be1a-cedfe48ed659-kube-api-access-5bl2s" (OuterVolumeSpecName: "kube-api-access-5bl2s") pod "e864f7ee-a354-4b50-be1a-cedfe48ed659" (UID: "e864f7ee-a354-4b50-be1a-cedfe48ed659"). InnerVolumeSpecName "kube-api-access-5bl2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:56:15.263247 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.263214    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bl2s\" (UniqueName: \"kubernetes.io/projected/e864f7ee-a354-4b50-be1a-cedfe48ed659-kube-api-access-5bl2s\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:56:15.263247 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.263241    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e864f7ee-a354-4b50-be1a-cedfe48ed659-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:56:15.263247 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.263251    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e864f7ee-a354-4b50-be1a-cedfe48ed659-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:56:15.445563 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.445537    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"]
Jul 22 02:56:15.452239 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:15.452212    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3c6d7b74-54cfc7966b-jpljc"]
Jul 22 02:56:16.868702 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:16.868666    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e864f7ee-a354-4b50-be1a-cedfe48ed659" path="/var/lib/kubelet/pods/e864f7ee-a354-4b50-be1a-cedfe48ed659/volumes"
Jul 22 02:56:23.486823 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.486786    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"]
Jul 22 02:56:23.489252 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.487123    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e864f7ee-a354-4b50-be1a-cedfe48ed659" containerName="maas-api"
Jul 22 02:56:23.489252 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.487134    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e864f7ee-a354-4b50-be1a-cedfe48ed659" containerName="maas-api"
Jul 22 02:56:23.489252 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.487202    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e864f7ee-a354-4b50-be1a-cedfe48ed659" containerName="maas-api"
Jul 22 02:56:23.490174 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.490159    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:23.492811 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.492790    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-113f7429-serving-cert\""
Jul 22 02:56:23.499991 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.499970    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"]
Jul 22 02:56:23.536296 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.536259    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2d3727-e269-4fba-a1b5-c85df7624661-openshift-service-ca\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:23.536484 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.536307    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:23.536484 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.536335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxgj\" (UniqueName: \"kubernetes.io/projected/bc2d3727-e269-4fba-a1b5-c85df7624661-kube-api-access-vkxgj\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:23.637636 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.637602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2d3727-e269-4fba-a1b5-c85df7624661-openshift-service-ca\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:23.637807 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.637643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:23.637807 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.637675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxgj\" (UniqueName: \"kubernetes.io/projected/bc2d3727-e269-4fba-a1b5-c85df7624661-kube-api-access-vkxgj\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:23.637807 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:56:23.637785    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-serving-cert: secret "maas-api-e2e-migrate-113f7429-serving-cert" not found
Jul 22 02:56:23.637909 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:56:23.637864    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls podName:bc2d3727-e269-4fba-a1b5-c85df7624661 nodeName:}" failed. No retries permitted until 2026-07-22 02:56:24.137846479 +0000 UTC m=+1673.818579503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls") pod "maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls" (UID: "bc2d3727-e269-4fba-a1b5-c85df7624661") : secret "maas-api-e2e-migrate-113f7429-serving-cert" not found
Jul 22 02:56:23.638364 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.638345    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2d3727-e269-4fba-a1b5-c85df7624661-openshift-service-ca\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:23.649243 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:23.649216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxgj\" (UniqueName: \"kubernetes.io/projected/bc2d3727-e269-4fba-a1b5-c85df7624661-kube-api-access-vkxgj\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:24.141344 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:24.141307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:24.143766 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:24.143747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls\") pod \"maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:24.401209 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:24.401100    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:24.535876 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:24.535846    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"]
Jul 22 02:56:24.539509 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:56:24.539479    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2d3727_e269_4fba_a1b5_c85df7624661.slice/crio-d4d769d98af803fe134f3e7aab4826e4cd01030418f73d9204b651c4b8c7ff7f WatchSource:0}: Error finding container d4d769d98af803fe134f3e7aab4826e4cd01030418f73d9204b651c4b8c7ff7f: Status 404 returned error can't find the container with id d4d769d98af803fe134f3e7aab4826e4cd01030418f73d9204b651c4b8c7ff7f
Jul 22 02:56:25.170358 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:25.170276    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls" event={"ID":"bc2d3727-e269-4fba-a1b5-c85df7624661","Type":"ContainerStarted","Data":"cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd"}
Jul 22 02:56:25.170358 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:25.170330    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls" event={"ID":"bc2d3727-e269-4fba-a1b5-c85df7624661","Type":"ContainerStarted","Data":"d4d769d98af803fe134f3e7aab4826e4cd01030418f73d9204b651c4b8c7ff7f"}
Jul 22 02:56:25.170536 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:25.170397    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:25.187142 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:25.187094    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls" podStartSLOduration=1.807862802 podStartE2EDuration="2.187076545s" totalImagesPullingTime="379.213743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:56:25.185873329 +0000 UTC m=+1674.866606375" watchObservedRunningTime="2026-07-22 02:56:25.187076545 +0000 UTC m=+1674.867809967"
Jul 22 02:56:31.181116 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:31.181066    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:58.919860 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:58.919771    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"]
Jul 22 02:56:58.920374 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:58.920019    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls" podUID="bc2d3727-e269-4fba-a1b5-c85df7624661" containerName="maas-api" containerID="cri-o://cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd" gracePeriod=30
Jul 22 02:56:59.166680 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.166656    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:59.273597 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.273566    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls\") pod \"bc2d3727-e269-4fba-a1b5-c85df7624661\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") "
Jul 22 02:56:59.273779 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.273609    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkxgj\" (UniqueName: \"kubernetes.io/projected/bc2d3727-e269-4fba-a1b5-c85df7624661-kube-api-access-vkxgj\") pod \"bc2d3727-e269-4fba-a1b5-c85df7624661\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") "
Jul 22 02:56:59.273829 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.273771    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2d3727-e269-4fba-a1b5-c85df7624661-openshift-service-ca\") pod \"bc2d3727-e269-4fba-a1b5-c85df7624661\" (UID: \"bc2d3727-e269-4fba-a1b5-c85df7624661\") "
Jul 22 02:56:59.274168 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.274126    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2d3727-e269-4fba-a1b5-c85df7624661-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc2d3727-e269-4fba-a1b5-c85df7624661" (UID: "bc2d3727-e269-4fba-a1b5-c85df7624661"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:56:59.275815 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.275795    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc2d3727-e269-4fba-a1b5-c85df7624661" (UID: "bc2d3727-e269-4fba-a1b5-c85df7624661"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:56:59.275879 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.275837    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2d3727-e269-4fba-a1b5-c85df7624661-kube-api-access-vkxgj" (OuterVolumeSpecName: "kube-api-access-vkxgj") pod "bc2d3727-e269-4fba-a1b5-c85df7624661" (UID: "bc2d3727-e269-4fba-a1b5-c85df7624661"). InnerVolumeSpecName "kube-api-access-vkxgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:56:59.319134 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.319093    2583 generic.go:356] "Generic (PLEG): container finished" podID="bc2d3727-e269-4fba-a1b5-c85df7624661" containerID="cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd" exitCode=0
Jul 22 02:56:59.319308 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.319138    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"
Jul 22 02:56:59.319308 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.319181    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls" event={"ID":"bc2d3727-e269-4fba-a1b5-c85df7624661","Type":"ContainerDied","Data":"cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd"}
Jul 22 02:56:59.319308 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.319232    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls" event={"ID":"bc2d3727-e269-4fba-a1b5-c85df7624661","Type":"ContainerDied","Data":"d4d769d98af803fe134f3e7aab4826e4cd01030418f73d9204b651c4b8c7ff7f"}
Jul 22 02:56:59.319308 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.319253    2583 scope.go:117] "RemoveContainer" containerID="cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd"
Jul 22 02:56:59.328127 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.328104    2583 scope.go:117] "RemoveContainer" containerID="cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd"
Jul 22 02:56:59.328414 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:56:59.328394    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd\": container with ID starting with cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd not found: ID does not exist" containerID="cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd"
Jul 22 02:56:59.328493 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.328424    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd"} err="failed to get container status \"cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd\": rpc error: code = NotFound desc = could not find container \"cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd\": container with ID starting with cde7ec7c7121ad0477ede505ed4ec05768dbd5d31e6acd91db206dd2cffe8afd not found: ID does not exist"
Jul 22 02:56:59.339850 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.339823    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"]
Jul 22 02:56:59.342351 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.342330    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-113f7429-5df849ff46-jl5ls"]
Jul 22 02:56:59.375287 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.375257    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc2d3727-e269-4fba-a1b5-c85df7624661-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:56:59.375287 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.375285    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxgj\" (UniqueName: \"kubernetes.io/projected/bc2d3727-e269-4fba-a1b5-c85df7624661-kube-api-access-vkxgj\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:56:59.375434 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:56:59.375295    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2d3727-e269-4fba-a1b5-c85df7624661-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:57:00.868565 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:00.868530    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc2d3727-e269-4fba-a1b5-c85df7624661" path="/var/lib/kubelet/pods/bc2d3727-e269-4fba-a1b5-c85df7624661/volumes"
Jul 22 02:57:07.159675 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.159633    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"]
Jul 22 02:57:07.160166 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.160111    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2d3727-e269-4fba-a1b5-c85df7624661" containerName="maas-api"
Jul 22 02:57:07.160166 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.160127    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2d3727-e269-4fba-a1b5-c85df7624661" containerName="maas-api"
Jul 22 02:57:07.160280 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.160184    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2d3727-e269-4fba-a1b5-c85df7624661" containerName="maas-api"
Jul 22 02:57:07.163396 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.163373    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.165985 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.165965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-49fcd44d-serving-cert\""
Jul 22 02:57:07.175178 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.175150    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"]
Jul 22 02:57:07.236283 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.236253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.236283 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.236291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518ce1a-956b-4ab1-8c34-0139ff67aee5-openshift-service-ca\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.236509 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.236424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6msz4\" (UniqueName: \"kubernetes.io/projected/c518ce1a-956b-4ab1-8c34-0139ff67aee5-kube-api-access-6msz4\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.337168 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.337120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6msz4\" (UniqueName: \"kubernetes.io/projected/c518ce1a-956b-4ab1-8c34-0139ff67aee5-kube-api-access-6msz4\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.337327 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.337179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.337327 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.337206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518ce1a-956b-4ab1-8c34-0139ff67aee5-openshift-service-ca\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.337399 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:57:07.337335    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-serving-cert: secret "maas-api-e2e-ait-49fcd44d-serving-cert" not found
Jul 22 02:57:07.337438 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:57:07.337396    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls podName:c518ce1a-956b-4ab1-8c34-0139ff67aee5 nodeName:}" failed. No retries permitted until 2026-07-22 02:57:07.837381223 +0000 UTC m=+1717.518114247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls") pod "maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c" (UID: "c518ce1a-956b-4ab1-8c34-0139ff67aee5") : secret "maas-api-e2e-ait-49fcd44d-serving-cert" not found
Jul 22 02:57:07.337852 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.337834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518ce1a-956b-4ab1-8c34-0139ff67aee5-openshift-service-ca\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.352484 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.352459    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6msz4\" (UniqueName: \"kubernetes.io/projected/c518ce1a-956b-4ab1-8c34-0139ff67aee5-kube-api-access-6msz4\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.842672 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.842632    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:07.845257 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:07.845235    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls\") pod \"maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:08.073574 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:08.073540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:08.207873 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:08.207851    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"]
Jul 22 02:57:08.210405 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:57:08.210377    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc518ce1a_956b_4ab1_8c34_0139ff67aee5.slice/crio-8650f05d56530c42972497a48feb93520c4138c4e3d698e57faa8aa841a69ec0 WatchSource:0}: Error finding container 8650f05d56530c42972497a48feb93520c4138c4e3d698e57faa8aa841a69ec0: Status 404 returned error can't find the container with id 8650f05d56530c42972497a48feb93520c4138c4e3d698e57faa8aa841a69ec0
Jul 22 02:57:08.361692 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:08.361662    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c" event={"ID":"c518ce1a-956b-4ab1-8c34-0139ff67aee5","Type":"ContainerStarted","Data":"8650f05d56530c42972497a48feb93520c4138c4e3d698e57faa8aa841a69ec0"}
Jul 22 02:57:09.368088 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:09.368053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c" event={"ID":"c518ce1a-956b-4ab1-8c34-0139ff67aee5","Type":"ContainerStarted","Data":"1bcc3a8f78d0145137b116f8f132dc8da39866bec9e10add0149926cad6152c4"}
Jul 22 02:57:09.368556 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:09.368197    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:09.406055 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:09.405983    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c" podStartSLOduration=1.8864885230000001 podStartE2EDuration="2.405966956s" totalImagesPullingTime="519.478433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:57:09.40497037 +0000 UTC m=+1719.085703417" watchObservedRunningTime="2026-07-22 02:57:09.405966956 +0000 UTC m=+1719.086700001"
Jul 22 02:57:15.378645 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:15.378618    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:43.343420 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.343389    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"]
Jul 22 02:57:43.343902 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.343650    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c" podUID="c518ce1a-956b-4ab1-8c34-0139ff67aee5" containerName="maas-api" containerID="cri-o://1bcc3a8f78d0145137b116f8f132dc8da39866bec9e10add0149926cad6152c4" gracePeriod=30
Jul 22 02:57:43.530191 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.530158    2583 generic.go:356] "Generic (PLEG): container finished" podID="c518ce1a-956b-4ab1-8c34-0139ff67aee5" containerID="1bcc3a8f78d0145137b116f8f132dc8da39866bec9e10add0149926cad6152c4" exitCode=0
Jul 22 02:57:43.530339 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.530212    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c" event={"ID":"c518ce1a-956b-4ab1-8c34-0139ff67aee5","Type":"ContainerDied","Data":"1bcc3a8f78d0145137b116f8f132dc8da39866bec9e10add0149926cad6152c4"}
Jul 22 02:57:43.595949 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.595865    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:43.769183 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.769142    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls\") pod \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") "
Jul 22 02:57:43.769369 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.769230    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518ce1a-956b-4ab1-8c34-0139ff67aee5-openshift-service-ca\") pod \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") "
Jul 22 02:57:43.769369 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.769297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6msz4\" (UniqueName: \"kubernetes.io/projected/c518ce1a-956b-4ab1-8c34-0139ff67aee5-kube-api-access-6msz4\") pod \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\" (UID: \"c518ce1a-956b-4ab1-8c34-0139ff67aee5\") "
Jul 22 02:57:43.769578 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.769554    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c518ce1a-956b-4ab1-8c34-0139ff67aee5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c518ce1a-956b-4ab1-8c34-0139ff67aee5" (UID: "c518ce1a-956b-4ab1-8c34-0139ff67aee5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:57:43.771500 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.771465    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c518ce1a-956b-4ab1-8c34-0139ff67aee5-kube-api-access-6msz4" (OuterVolumeSpecName: "kube-api-access-6msz4") pod "c518ce1a-956b-4ab1-8c34-0139ff67aee5" (UID: "c518ce1a-956b-4ab1-8c34-0139ff67aee5"). InnerVolumeSpecName "kube-api-access-6msz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:57:43.771500 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.771477    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c518ce1a-956b-4ab1-8c34-0139ff67aee5" (UID: "c518ce1a-956b-4ab1-8c34-0139ff67aee5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:57:43.870971 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.870890    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c518ce1a-956b-4ab1-8c34-0139ff67aee5-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:57:43.870971 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.870920    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c518ce1a-956b-4ab1-8c34-0139ff67aee5-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:57:43.870971 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:43.870931    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6msz4\" (UniqueName: \"kubernetes.io/projected/c518ce1a-956b-4ab1-8c34-0139ff67aee5-kube-api-access-6msz4\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:57:44.535770 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:44.535734    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c" event={"ID":"c518ce1a-956b-4ab1-8c34-0139ff67aee5","Type":"ContainerDied","Data":"8650f05d56530c42972497a48feb93520c4138c4e3d698e57faa8aa841a69ec0"}
Jul 22 02:57:44.536215 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:44.535793    2583 scope.go:117] "RemoveContainer" containerID="1bcc3a8f78d0145137b116f8f132dc8da39866bec9e10add0149926cad6152c4"
Jul 22 02:57:44.536215 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:44.535749    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"
Jul 22 02:57:44.557792 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:44.557761    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"]
Jul 22 02:57:44.562079 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:44.562048    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-49fcd44d-7756f94756-tzl9c"]
Jul 22 02:57:44.868323 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:44.868244    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c518ce1a-956b-4ab1-8c34-0139ff67aee5" path="/var/lib/kubelet/pods/c518ce1a-956b-4ab1-8c34-0139ff67aee5/volumes"
Jul 22 02:57:52.899952 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:52.899916    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"]
Jul 22 02:57:52.900493 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:52.900474    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c518ce1a-956b-4ab1-8c34-0139ff67aee5" containerName="maas-api"
Jul 22 02:57:52.900537 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:52.900497    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c518ce1a-956b-4ab1-8c34-0139ff67aee5" containerName="maas-api"
Jul 22 02:57:52.900577 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:52.900572    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c518ce1a-956b-4ab1-8c34-0139ff67aee5" containerName="maas-api"
Jul 22 02:57:52.905254 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:52.905238    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:52.907574 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:52.907551    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e4d739e2-serving-cert\""
Jul 22 02:57:52.915762 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:52.915731    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"]
Jul 22 02:57:53.051379 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.051343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-openshift-service-ca\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.051379 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.051382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2qqx\" (UniqueName: \"kubernetes.io/projected/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-kube-api-access-b2qqx\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.051608 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.051487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-maas-api-tls\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.152861 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.152759    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-openshift-service-ca\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.152861 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.152807    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2qqx\" (UniqueName: \"kubernetes.io/projected/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-kube-api-access-b2qqx\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.153110 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.152887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-maas-api-tls\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.153586 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.153559    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-openshift-service-ca\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.155577 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.155556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-maas-api-tls\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.161960 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.161934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2qqx\" (UniqueName: \"kubernetes.io/projected/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-kube-api-access-b2qqx\") pod \"maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.215451 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.215419    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:53.360065 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.360008    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"]
Jul 22 02:57:53.363066 ip-10-0-133-30 kubenswrapper[2583]: W0722 02:57:53.363002    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8593f44_2573_4b1d_b8a5_cc6bd1a10d4b.slice/crio-5e30ec4b4af507c5c83d66897c9759703f1ea5038158f25bfc706bc5bc49fa9d WatchSource:0}: Error finding container 5e30ec4b4af507c5c83d66897c9759703f1ea5038158f25bfc706bc5bc49fa9d: Status 404 returned error can't find the container with id 5e30ec4b4af507c5c83d66897c9759703f1ea5038158f25bfc706bc5bc49fa9d
Jul 22 02:57:53.585238 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:53.585200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m" event={"ID":"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b","Type":"ContainerStarted","Data":"5e30ec4b4af507c5c83d66897c9759703f1ea5038158f25bfc706bc5bc49fa9d"}
Jul 22 02:57:54.591163 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:54.591121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m" event={"ID":"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b","Type":"ContainerStarted","Data":"dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067"}
Jul 22 02:57:54.591560 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:54.591195    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:57:54.608138 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:57:54.608091    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m" podStartSLOduration=2.15773911 podStartE2EDuration="2.608075462s" totalImagesPullingTime="450.336352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 02:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 02:57:54.606762375 +0000 UTC m=+1764.287495434" watchObservedRunningTime="2026-07-22 02:57:54.608075462 +0000 UTC m=+1764.288808508"
Jul 22 02:58:00.600425 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:00.600392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:58:28.233614 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.233572    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"]
Jul 22 02:58:28.234214 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.233846    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m" podUID="d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" containerName="maas-api" containerID="cri-o://dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067" gracePeriod=30
Jul 22 02:58:28.486643 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.486575    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:58:28.566353 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.566316    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2qqx\" (UniqueName: \"kubernetes.io/projected/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-kube-api-access-b2qqx\") pod \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") "
Jul 22 02:58:28.566546 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.566431    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-openshift-service-ca\") pod \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") "
Jul 22 02:58:28.566546 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.566463    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-maas-api-tls\") pod \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\" (UID: \"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b\") "
Jul 22 02:58:28.566803 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.566779    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" (UID: "d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 02:58:28.568677 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.568639    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" (UID: "d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 02:58:28.568818 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.568700    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-kube-api-access-b2qqx" (OuterVolumeSpecName: "kube-api-access-b2qqx") pod "d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" (UID: "d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b"). InnerVolumeSpecName "kube-api-access-b2qqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 02:58:28.667576 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.667537    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:58:28.667576 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.667570    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:58:28.667576 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.667579    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2qqx\" (UniqueName: \"kubernetes.io/projected/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b-kube-api-access-b2qqx\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 02:58:28.745579 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.745499    2583 generic.go:356] "Generic (PLEG): container finished" podID="d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" containerID="dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067" exitCode=0
Jul 22 02:58:28.745579 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.745548    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"
Jul 22 02:58:28.745748 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.745590    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m" event={"ID":"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b","Type":"ContainerDied","Data":"dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067"}
Jul 22 02:58:28.745748 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.745642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m" event={"ID":"d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b","Type":"ContainerDied","Data":"5e30ec4b4af507c5c83d66897c9759703f1ea5038158f25bfc706bc5bc49fa9d"}
Jul 22 02:58:28.745748 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.745664    2583 scope.go:117] "RemoveContainer" containerID="dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067"
Jul 22 02:58:28.754211 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.754195    2583 scope.go:117] "RemoveContainer" containerID="dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067"
Jul 22 02:58:28.754455 ip-10-0-133-30 kubenswrapper[2583]: E0722 02:58:28.754433    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067\": container with ID starting with dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067 not found: ID does not exist" containerID="dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067"
Jul 22 02:58:28.754521 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.754468    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067"} err="failed to get container status \"dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067\": rpc error: code = NotFound desc = could not find container \"dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067\": container with ID starting with dd8911f5c9f2f2fd1979bd8c2198016045ef5601070a3c5ab6a2b5f65191b067 not found: ID does not exist"
Jul 22 02:58:28.765421 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.765395    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"]
Jul 22 02:58:28.769354 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.769336    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e4d739e2-7f9dbfdfdb-gj27m"]
Jul 22 02:58:28.868995 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:28.868964    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" path="/var/lib/kubelet/pods/d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b/volumes"
Jul 22 02:58:30.841453 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:30.841427    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 02:58:30.845045 ip-10-0-133-30 kubenswrapper[2583]: I0722 02:58:30.845006    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 03:00:19.682827 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.682795    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"]
Jul 22 03:00:19.683539 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.683497    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" containerName="maas-api"
Jul 22 03:00:19.683539 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.683528    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" containerName="maas-api"
Jul 22 03:00:19.683711 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.683608    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8593f44-2573-4b1d-b8a5-cc6bd1a10d4b" containerName="maas-api"
Jul 22 03:00:19.686531 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.686512    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:19.690675 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.690655    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-08162364-serving-cert\""
Jul 22 03:00:19.701850 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.701814    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"]
Jul 22 03:00:19.851813 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.851776    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkksn\" (UniqueName: \"kubernetes.io/projected/81e8317d-3e78-489c-8d12-4f39928ae600-kube-api-access-xkksn\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:19.851813 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.851819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:19.852023 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.851859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e8317d-3e78-489c-8d12-4f39928ae600-openshift-service-ca\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:19.953074 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.952975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e8317d-3e78-489c-8d12-4f39928ae600-openshift-service-ca\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:19.953203 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.953105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkksn\" (UniqueName: \"kubernetes.io/projected/81e8317d-3e78-489c-8d12-4f39928ae600-kube-api-access-xkksn\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:19.953203 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.953138    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:19.953279 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:00:19.953261    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-08162364-serving-cert: secret "maas-api-e2e-mt-08162364-serving-cert" not found
Jul 22 03:00:19.953358 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:00:19.953346    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls podName:81e8317d-3e78-489c-8d12-4f39928ae600 nodeName:}" failed. No retries permitted until 2026-07-22 03:00:20.453325108 +0000 UTC m=+1910.134058131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls") pod "maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz" (UID: "81e8317d-3e78-489c-8d12-4f39928ae600") : secret "maas-api-e2e-mt-08162364-serving-cert" not found
Jul 22 03:00:19.953721 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.953700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e8317d-3e78-489c-8d12-4f39928ae600-openshift-service-ca\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:19.966175 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:19.966146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkksn\" (UniqueName: \"kubernetes.io/projected/81e8317d-3e78-489c-8d12-4f39928ae600-kube-api-access-xkksn\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:20.457776 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:20.457742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:20.460243 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:20.460223    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls\") pod \"maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:20.596467 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:20.596433    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:20.731372 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:20.731292    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"]
Jul 22 03:00:20.733797 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:00:20.733773    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e8317d_3e78_489c_8d12_4f39928ae600.slice/crio-f84947e9258208370ddfef2257bb0c0fcd49f758f32c9ada6aa086d8cb8a784e WatchSource:0}: Error finding container f84947e9258208370ddfef2257bb0c0fcd49f758f32c9ada6aa086d8cb8a784e: Status 404 returned error can't find the container with id f84947e9258208370ddfef2257bb0c0fcd49f758f32c9ada6aa086d8cb8a784e
Jul 22 03:00:21.256817 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:21.256788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz" event={"ID":"81e8317d-3e78-489c-8d12-4f39928ae600","Type":"ContainerStarted","Data":"f84947e9258208370ddfef2257bb0c0fcd49f758f32c9ada6aa086d8cb8a784e"}
Jul 22 03:00:22.262314 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:22.262270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz" event={"ID":"81e8317d-3e78-489c-8d12-4f39928ae600","Type":"ContainerStarted","Data":"aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f"}
Jul 22 03:00:22.262727 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:22.262353    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:22.285790 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:22.285737    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz" podStartSLOduration=2.813855378 podStartE2EDuration="3.28572003s" totalImagesPullingTime="471.864652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:00:22.283717665 +0000 UTC m=+1911.964450716" watchObservedRunningTime="2026-07-22 03:00:22.28572003 +0000 UTC m=+1911.966453076"
Jul 22 03:00:28.273814 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:28.273778    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:42.013916 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.013877    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"]
Jul 22 03:00:42.014498 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.014138    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz" podUID="81e8317d-3e78-489c-8d12-4f39928ae600" containerName="maas-api" containerID="cri-o://aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f" gracePeriod=30
Jul 22 03:00:42.260928 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.260900    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:42.351773 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.351688    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls\") pod \"81e8317d-3e78-489c-8d12-4f39928ae600\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") "
Jul 22 03:00:42.351942 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.351816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e8317d-3e78-489c-8d12-4f39928ae600-openshift-service-ca\") pod \"81e8317d-3e78-489c-8d12-4f39928ae600\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") "
Jul 22 03:00:42.351942 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.351896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkksn\" (UniqueName: \"kubernetes.io/projected/81e8317d-3e78-489c-8d12-4f39928ae600-kube-api-access-xkksn\") pod \"81e8317d-3e78-489c-8d12-4f39928ae600\" (UID: \"81e8317d-3e78-489c-8d12-4f39928ae600\") "
Jul 22 03:00:42.352513 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.352306    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e8317d-3e78-489c-8d12-4f39928ae600-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81e8317d-3e78-489c-8d12-4f39928ae600" (UID: "81e8317d-3e78-489c-8d12-4f39928ae600"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:00:42.355055 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.354474    2583 generic.go:356] "Generic (PLEG): container finished" podID="81e8317d-3e78-489c-8d12-4f39928ae600" containerID="aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f" exitCode=0
Jul 22 03:00:42.355055 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.354585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz" event={"ID":"81e8317d-3e78-489c-8d12-4f39928ae600","Type":"ContainerDied","Data":"aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f"}
Jul 22 03:00:42.355055 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.354635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz" event={"ID":"81e8317d-3e78-489c-8d12-4f39928ae600","Type":"ContainerDied","Data":"f84947e9258208370ddfef2257bb0c0fcd49f758f32c9ada6aa086d8cb8a784e"}
Jul 22 03:00:42.355055 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.354661    2583 scope.go:117] "RemoveContainer" containerID="aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f"
Jul 22 03:00:42.355055 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.354823    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"
Jul 22 03:00:42.355461 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.355434    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81e8317d-3e78-489c-8d12-4f39928ae600" (UID: "81e8317d-3e78-489c-8d12-4f39928ae600"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:00:42.355649 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.355610    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e8317d-3e78-489c-8d12-4f39928ae600-kube-api-access-xkksn" (OuterVolumeSpecName: "kube-api-access-xkksn") pod "81e8317d-3e78-489c-8d12-4f39928ae600" (UID: "81e8317d-3e78-489c-8d12-4f39928ae600"). InnerVolumeSpecName "kube-api-access-xkksn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:00:42.371849 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.371825    2583 scope.go:117] "RemoveContainer" containerID="aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f"
Jul 22 03:00:42.372198 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:00:42.372174    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f\": container with ID starting with aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f not found: ID does not exist" containerID="aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f"
Jul 22 03:00:42.372268 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.372208    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f"} err="failed to get container status \"aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f\": rpc error: code = NotFound desc = could not find container \"aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f\": container with ID starting with aa260eac8642ba70e7200cbf66d57e35663757fd5b929cfc8ae45c86d49f188f not found: ID does not exist"
Jul 22 03:00:42.453432 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.453390    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkksn\" (UniqueName: \"kubernetes.io/projected/81e8317d-3e78-489c-8d12-4f39928ae600-kube-api-access-xkksn\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:00:42.453432 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.453428    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81e8317d-3e78-489c-8d12-4f39928ae600-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:00:42.453432 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.453439    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e8317d-3e78-489c-8d12-4f39928ae600-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:00:42.676334 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.676302    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"]
Jul 22 03:00:42.679945 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.679918    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08162364-bbd6fb69c-zd8fz"]
Jul 22 03:00:42.868979 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:00:42.868946    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e8317d-3e78-489c-8d12-4f39928ae600" path="/var/lib/kubelet/pods/81e8317d-3e78-489c-8d12-4f39928ae600/volumes"
Jul 22 03:01:16.879923 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.879883    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"]
Jul 22 03:01:16.880388 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.880229    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e8317d-3e78-489c-8d12-4f39928ae600" containerName="maas-api"
Jul 22 03:01:16.880388 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.880240    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e8317d-3e78-489c-8d12-4f39928ae600" containerName="maas-api"
Jul 22 03:01:16.880388 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.880302    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e8317d-3e78-489c-8d12-4f39928ae600" containerName="maas-api"
Jul 22 03:01:16.883156 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.883139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:16.885516 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.885494    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ee84bc3c-serving-cert\""
Jul 22 03:01:16.894788 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.894763    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"]
Jul 22 03:01:16.958501 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.958466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nv5x\" (UniqueName: \"kubernetes.io/projected/0698ffed-e7b2-4814-9a14-2509f3c68ed8-kube-api-access-6nv5x\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:16.958501 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.958505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:16.958736 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:16.958609    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0698ffed-e7b2-4814-9a14-2509f3c68ed8-openshift-service-ca\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.059877 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.059844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nv5x\" (UniqueName: \"kubernetes.io/projected/0698ffed-e7b2-4814-9a14-2509f3c68ed8-kube-api-access-6nv5x\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.060066 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.059890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.060066 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.060040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0698ffed-e7b2-4814-9a14-2509f3c68ed8-openshift-service-ca\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.060208 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:01:17.060135    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-serving-cert: secret "maas-api-e2e-mt-ee84bc3c-serving-cert" not found
Jul 22 03:01:17.060260 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:01:17.060219    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls podName:0698ffed-e7b2-4814-9a14-2509f3c68ed8 nodeName:}" failed. No retries permitted until 2026-07-22 03:01:17.560192946 +0000 UTC m=+1967.240925976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls") pod "maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz" (UID: "0698ffed-e7b2-4814-9a14-2509f3c68ed8") : secret "maas-api-e2e-mt-ee84bc3c-serving-cert" not found
Jul 22 03:01:17.060622 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.060603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0698ffed-e7b2-4814-9a14-2509f3c68ed8-openshift-service-ca\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.092162 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.092133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nv5x\" (UniqueName: \"kubernetes.io/projected/0698ffed-e7b2-4814-9a14-2509f3c68ed8-kube-api-access-6nv5x\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.564242 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.564201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.566768 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.566743    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls\") pod \"maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.793457 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.793400    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:17.929841 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.929673    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"]
Jul 22 03:01:17.932744 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:01:17.932712    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0698ffed_e7b2_4814_9a14_2509f3c68ed8.slice/crio-bf104c39457c79fdb8fddf710518a6aa9debc33ba7b1bd1c0bdaf17c1b5da39e WatchSource:0}: Error finding container bf104c39457c79fdb8fddf710518a6aa9debc33ba7b1bd1c0bdaf17c1b5da39e: Status 404 returned error can't find the container with id bf104c39457c79fdb8fddf710518a6aa9debc33ba7b1bd1c0bdaf17c1b5da39e
Jul 22 03:01:17.934262 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:17.934243    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 03:01:18.512426 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:18.512387    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz" event={"ID":"0698ffed-e7b2-4814-9a14-2509f3c68ed8","Type":"ContainerStarted","Data":"6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db"}
Jul 22 03:01:18.512601 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:18.512439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz" event={"ID":"0698ffed-e7b2-4814-9a14-2509f3c68ed8","Type":"ContainerStarted","Data":"bf104c39457c79fdb8fddf710518a6aa9debc33ba7b1bd1c0bdaf17c1b5da39e"}
Jul 22 03:01:18.512601 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:18.512519    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:18.530432 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:18.530374    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz" podStartSLOduration=2.145017084 podStartE2EDuration="2.530355511s" totalImagesPullingTime="385.338427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:01:18.528533133 +0000 UTC m=+1968.209266180" watchObservedRunningTime="2026-07-22 03:01:18.530355511 +0000 UTC m=+1968.211088556"
Jul 22 03:01:24.527800 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:24.527771    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:01:30.509419 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.509382    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"]
Jul 22 03:01:30.512301 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.512284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.515022 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.514998    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ee84bc3c\"/\"openshift-service-ca.crt\""
Jul 22 03:01:30.516120 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.516100    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ee84bc3c\"/\"e2e-lifecycle-model-ee84bc3c-kserve-self-signed-certs\""
Jul 22 03:01:30.516120 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.516114    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ee84bc3c\"/\"kube-root-ca.crt\""
Jul 22 03:01:30.516257 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.516177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ee84bc3c\"/\"default-dockercfg-5zdht\""
Jul 22 03:01:30.525596 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.525573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"]
Jul 22 03:01:30.579022 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.578992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-dshm\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.579022 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.579023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547125c1-ec2f-49aa-84dd-ea10417659c1-tls-certs\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.579284 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.579106    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-tmp-dir\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.579284 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.579128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwblk\" (UniqueName: \"kubernetes.io/projected/547125c1-ec2f-49aa-84dd-ea10417659c1-kube-api-access-dwblk\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.579284 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.579153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-model-cache\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.579284 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.579180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-home\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.679751 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.679713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-tmp-dir\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.679751 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.679754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwblk\" (UniqueName: \"kubernetes.io/projected/547125c1-ec2f-49aa-84dd-ea10417659c1-kube-api-access-dwblk\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.679978 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.679785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-model-cache\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.679978 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.679811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-home\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.679978 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.679891    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-dshm\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.679978 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.679913    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547125c1-ec2f-49aa-84dd-ea10417659c1-tls-certs\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.680277 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.680104    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-tmp-dir\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.680277 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.680192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-model-cache\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.680369 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.680307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-home\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.682200 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.682179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-dshm\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.686937 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.686919    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ee84bc3c\"/\"e2e-lifecycle-model-ee84bc3c-kserve-self-signed-certs\""
Jul 22 03:01:30.692833 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.692813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547125c1-ec2f-49aa-84dd-ea10417659c1-tls-certs\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.692934 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.692851    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ee84bc3c\"/\"kube-root-ca.crt\""
Jul 22 03:01:30.703213 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.703195    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ee84bc3c\"/\"openshift-service-ca.crt\""
Jul 22 03:01:30.714545 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.714513    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwblk\" (UniqueName: \"kubernetes.io/projected/547125c1-ec2f-49aa-84dd-ea10417659c1-kube-api-access-dwblk\") pod \"e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") " pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.826527 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.826447    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ee84bc3c\"/\"default-dockercfg-5zdht\""
Jul 22 03:01:30.831993 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.831958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:30.979761 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:30.979735    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"]
Jul 22 03:01:30.981361 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:01:30.981334    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod547125c1_ec2f_49aa_84dd_ea10417659c1.slice/crio-1fbccf93ab9a3d78949fd8c411ce04a355d086d127c7284cdfca84e4dd09800e WatchSource:0}: Error finding container 1fbccf93ab9a3d78949fd8c411ce04a355d086d127c7284cdfca84e4dd09800e: Status 404 returned error can't find the container with id 1fbccf93ab9a3d78949fd8c411ce04a355d086d127c7284cdfca84e4dd09800e
Jul 22 03:01:31.576558 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:31.576522    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb" event={"ID":"547125c1-ec2f-49aa-84dd-ea10417659c1","Type":"ContainerStarted","Data":"0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66"}
Jul 22 03:01:31.576558 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:31.576565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb" event={"ID":"547125c1-ec2f-49aa-84dd-ea10417659c1","Type":"ContainerStarted","Data":"1fbccf93ab9a3d78949fd8c411ce04a355d086d127c7284cdfca84e4dd09800e"}
Jul 22 03:01:31.594015 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:31.593965    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb" podStartSLOduration=1.59395057 podStartE2EDuration="1.59395057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:01:31.592297292 +0000 UTC m=+1981.273030337" watchObservedRunningTime="2026-07-22 03:01:31.59395057 +0000 UTC m=+1981.274683616"
Jul 22 03:01:40.832460 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:40.832415    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:40.832460 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:40.832467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:40.845157 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:40.845127    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:01:41.631016 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:01:41.630991    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:02:04.877683 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:04.877653    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"]
Jul 22 03:02:04.878164 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:04.877858    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz" podUID="0698ffed-e7b2-4814-9a14-2509f3c68ed8" containerName="maas-api" containerID="cri-o://6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db" gracePeriod=30
Jul 22 03:02:05.121474 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.121447    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:02:05.196538 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.196448    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls\") pod \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") "
Jul 22 03:02:05.196538 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.196503    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nv5x\" (UniqueName: \"kubernetes.io/projected/0698ffed-e7b2-4814-9a14-2509f3c68ed8-kube-api-access-6nv5x\") pod \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") "
Jul 22 03:02:05.196726 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.196594    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0698ffed-e7b2-4814-9a14-2509f3c68ed8-openshift-service-ca\") pod \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\" (UID: \"0698ffed-e7b2-4814-9a14-2509f3c68ed8\") "
Jul 22 03:02:05.196964 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.196930    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0698ffed-e7b2-4814-9a14-2509f3c68ed8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0698ffed-e7b2-4814-9a14-2509f3c68ed8" (UID: "0698ffed-e7b2-4814-9a14-2509f3c68ed8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:02:05.198714 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.198683    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0698ffed-e7b2-4814-9a14-2509f3c68ed8" (UID: "0698ffed-e7b2-4814-9a14-2509f3c68ed8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:02:05.198714 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.198694    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0698ffed-e7b2-4814-9a14-2509f3c68ed8-kube-api-access-6nv5x" (OuterVolumeSpecName: "kube-api-access-6nv5x") pod "0698ffed-e7b2-4814-9a14-2509f3c68ed8" (UID: "0698ffed-e7b2-4814-9a14-2509f3c68ed8"). InnerVolumeSpecName "kube-api-access-6nv5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:02:05.297665 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.297619    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nv5x\" (UniqueName: \"kubernetes.io/projected/0698ffed-e7b2-4814-9a14-2509f3c68ed8-kube-api-access-6nv5x\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:05.297665 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.297654    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0698ffed-e7b2-4814-9a14-2509f3c68ed8-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:05.297665 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.297664    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0698ffed-e7b2-4814-9a14-2509f3c68ed8-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:05.726270 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.726230    2583 generic.go:356] "Generic (PLEG): container finished" podID="0698ffed-e7b2-4814-9a14-2509f3c68ed8" containerID="6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db" exitCode=0
Jul 22 03:02:05.726451 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.726275    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"
Jul 22 03:02:05.726451 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.726307    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz" event={"ID":"0698ffed-e7b2-4814-9a14-2509f3c68ed8","Type":"ContainerDied","Data":"6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db"}
Jul 22 03:02:05.726451 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.726356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz" event={"ID":"0698ffed-e7b2-4814-9a14-2509f3c68ed8","Type":"ContainerDied","Data":"bf104c39457c79fdb8fddf710518a6aa9debc33ba7b1bd1c0bdaf17c1b5da39e"}
Jul 22 03:02:05.726451 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.726378    2583 scope.go:117] "RemoveContainer" containerID="6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db"
Jul 22 03:02:05.735299 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.735273    2583 scope.go:117] "RemoveContainer" containerID="6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db"
Jul 22 03:02:05.735600 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:02:05.735578    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db\": container with ID starting with 6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db not found: ID does not exist" containerID="6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db"
Jul 22 03:02:05.735672 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.735615    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db"} err="failed to get container status \"6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db\": rpc error: code = NotFound desc = could not find container \"6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db\": container with ID starting with 6761fedff674dbd7e6028e0696a614793937efc1647b2c108114072b1ddd19db not found: ID does not exist"
Jul 22 03:02:05.751447 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.751417    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"]
Jul 22 03:02:05.756238 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:05.756215    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ee84bc3c-564686999b-gdfcz"]
Jul 22 03:02:06.873821 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:06.873787    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0698ffed-e7b2-4814-9a14-2509f3c68ed8" path="/var/lib/kubelet/pods/0698ffed-e7b2-4814-9a14-2509f3c68ed8/volumes"
Jul 22 03:02:16.722842 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:16.722808    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"]
Jul 22 03:02:31.743415 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:31.743371    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb" podUID="547125c1-ec2f-49aa-84dd-ea10417659c1" containerName="main" containerID="cri-o://0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66" gracePeriod=45
Jul 22 03:02:31.988517 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:31.988494    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:02:32.147563 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147469    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547125c1-ec2f-49aa-84dd-ea10417659c1-tls-certs\") pod \"547125c1-ec2f-49aa-84dd-ea10417659c1\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") "
Jul 22 03:02:32.147563 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147537    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-dshm\") pod \"547125c1-ec2f-49aa-84dd-ea10417659c1\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") "
Jul 22 03:02:32.147798 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147588    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-tmp-dir\") pod \"547125c1-ec2f-49aa-84dd-ea10417659c1\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") "
Jul 22 03:02:32.147798 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147608    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-home\") pod \"547125c1-ec2f-49aa-84dd-ea10417659c1\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") "
Jul 22 03:02:32.147798 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-model-cache\") pod \"547125c1-ec2f-49aa-84dd-ea10417659c1\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") "
Jul 22 03:02:32.147798 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147669    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwblk\" (UniqueName: \"kubernetes.io/projected/547125c1-ec2f-49aa-84dd-ea10417659c1-kube-api-access-dwblk\") pod \"547125c1-ec2f-49aa-84dd-ea10417659c1\" (UID: \"547125c1-ec2f-49aa-84dd-ea10417659c1\") "
Jul 22 03:02:32.147984 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-home" (OuterVolumeSpecName: "home") pod "547125c1-ec2f-49aa-84dd-ea10417659c1" (UID: "547125c1-ec2f-49aa-84dd-ea10417659c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:02:32.147984 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147879    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "547125c1-ec2f-49aa-84dd-ea10417659c1" (UID: "547125c1-ec2f-49aa-84dd-ea10417659c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:02:32.147984 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.147896    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "547125c1-ec2f-49aa-84dd-ea10417659c1" (UID: "547125c1-ec2f-49aa-84dd-ea10417659c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:02:32.150000 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.149959    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-dshm" (OuterVolumeSpecName: "dshm") pod "547125c1-ec2f-49aa-84dd-ea10417659c1" (UID: "547125c1-ec2f-49aa-84dd-ea10417659c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:02:32.150134 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.150051    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/547125c1-ec2f-49aa-84dd-ea10417659c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "547125c1-ec2f-49aa-84dd-ea10417659c1" (UID: "547125c1-ec2f-49aa-84dd-ea10417659c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:02:32.150134 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.150065    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547125c1-ec2f-49aa-84dd-ea10417659c1-kube-api-access-dwblk" (OuterVolumeSpecName: "kube-api-access-dwblk") pod "547125c1-ec2f-49aa-84dd-ea10417659c1" (UID: "547125c1-ec2f-49aa-84dd-ea10417659c1"). InnerVolumeSpecName "kube-api-access-dwblk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:02:32.248325 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.248288    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-tmp-dir\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:32.248325 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.248314    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-home\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:32.248325 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.248322    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-model-cache\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:32.248325 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.248332    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwblk\" (UniqueName: \"kubernetes.io/projected/547125c1-ec2f-49aa-84dd-ea10417659c1-kube-api-access-dwblk\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:32.248585 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.248341    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/547125c1-ec2f-49aa-84dd-ea10417659c1-tls-certs\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:32.248585 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.248350    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/547125c1-ec2f-49aa-84dd-ea10417659c1-dshm\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:02:32.850877 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.850842    2583 generic.go:356] "Generic (PLEG): container finished" podID="547125c1-ec2f-49aa-84dd-ea10417659c1" containerID="0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66" exitCode=0
Jul 22 03:02:32.851408 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.850892    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"
Jul 22 03:02:32.851408 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.850906    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb" event={"ID":"547125c1-ec2f-49aa-84dd-ea10417659c1","Type":"ContainerDied","Data":"0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66"}
Jul 22 03:02:32.851408 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.850940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb" event={"ID":"547125c1-ec2f-49aa-84dd-ea10417659c1","Type":"ContainerDied","Data":"1fbccf93ab9a3d78949fd8c411ce04a355d086d127c7284cdfca84e4dd09800e"}
Jul 22 03:02:32.851408 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.850957    2583 scope.go:117] "RemoveContainer" containerID="0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66"
Jul 22 03:02:32.859852 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.859830    2583 scope.go:117] "RemoveContainer" containerID="0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66"
Jul 22 03:02:32.860140 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:02:32.860116    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66\": container with ID starting with 0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66 not found: ID does not exist" containerID="0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66"
Jul 22 03:02:32.860239 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.860148    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66"} err="failed to get container status \"0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66\": rpc error: code = NotFound desc = could not find container \"0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66\": container with ID starting with 0112c770b8b975ba7c9e0f563d5a738559531fe8ecfd1badd1f0fb6dacbc3c66 not found: ID does not exist"
Jul 22 03:02:32.878456 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.874401    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"]
Jul 22 03:02:32.881631 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:32.881600    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ee84bc3c/e2e-lifecycle-model-ee84bc3c-kserve-bfcf4d6d7-qcfbb"]
Jul 22 03:02:34.869291 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:02:34.869256    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="547125c1-ec2f-49aa-84dd-ea10417659c1" path="/var/lib/kubelet/pods/547125c1-ec2f-49aa-84dd-ea10417659c1/volumes"
Jul 22 03:03:30.881260 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:30.881230    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 03:03:30.881765 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:30.881384    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 03:03:59.691184 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.691146    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"]
Jul 22 03:03:59.691683 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.691477    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0698ffed-e7b2-4814-9a14-2509f3c68ed8" containerName="maas-api"
Jul 22 03:03:59.691683 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.691488    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0698ffed-e7b2-4814-9a14-2509f3c68ed8" containerName="maas-api"
Jul 22 03:03:59.691683 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.691500    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="547125c1-ec2f-49aa-84dd-ea10417659c1" containerName="main"
Jul 22 03:03:59.691683 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.691505    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="547125c1-ec2f-49aa-84dd-ea10417659c1" containerName="main"
Jul 22 03:03:59.691683 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.691581    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="547125c1-ec2f-49aa-84dd-ea10417659c1" containerName="main"
Jul 22 03:03:59.691683 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.691592    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0698ffed-e7b2-4814-9a14-2509f3c68ed8" containerName="maas-api"
Jul 22 03:03:59.694646 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.694628    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.697499 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.697476    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-317968-serving-cert\""
Jul 22 03:03:59.712630 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.712606    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"]
Jul 22 03:03:59.838336 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.838300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kznw6\" (UniqueName: \"kubernetes.io/projected/6205d85e-ba11-40c3-94db-5dab7408af73-kube-api-access-kznw6\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.838336 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.838344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6205d85e-ba11-40c3-94db-5dab7408af73-maas-api-tls\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.838561 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.838422    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6205d85e-ba11-40c3-94db-5dab7408af73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.939347 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.939317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6205d85e-ba11-40c3-94db-5dab7408af73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.939508 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.939385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kznw6\" (UniqueName: \"kubernetes.io/projected/6205d85e-ba11-40c3-94db-5dab7408af73-kube-api-access-kznw6\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.939508 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.939419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6205d85e-ba11-40c3-94db-5dab7408af73-maas-api-tls\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.940089 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.940068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6205d85e-ba11-40c3-94db-5dab7408af73-openshift-service-ca\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.942016 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.941972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6205d85e-ba11-40c3-94db-5dab7408af73-maas-api-tls\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:03:59.961127 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:03:59.961090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kznw6\" (UniqueName: \"kubernetes.io/projected/6205d85e-ba11-40c3-94db-5dab7408af73-kube-api-access-kznw6\") pod \"maas-api-e2e-inf-a-317968-64c869fd79-w692v\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:04:00.009896 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:00.009853    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:04:00.161050 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:00.161011    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"]
Jul 22 03:04:00.163426 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:04:00.163393    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6205d85e_ba11_40c3_94db_5dab7408af73.slice/crio-daa06e051840a25493f1d91f13302e1d1ffdf343819bdb88db19aa03830ba63e WatchSource:0}: Error finding container daa06e051840a25493f1d91f13302e1d1ffdf343819bdb88db19aa03830ba63e: Status 404 returned error can't find the container with id daa06e051840a25493f1d91f13302e1d1ffdf343819bdb88db19aa03830ba63e
Jul 22 03:04:00.229643 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:00.229609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v" event={"ID":"6205d85e-ba11-40c3-94db-5dab7408af73","Type":"ContainerStarted","Data":"daa06e051840a25493f1d91f13302e1d1ffdf343819bdb88db19aa03830ba63e"}
Jul 22 03:04:01.237178 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:01.237140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v" event={"ID":"6205d85e-ba11-40c3-94db-5dab7408af73","Type":"ContainerStarted","Data":"a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd"}
Jul 22 03:04:01.237679 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:01.237635    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:04:01.264468 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:01.264407    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v" podStartSLOduration=1.8919918230000001 podStartE2EDuration="2.264389459s" totalImagesPullingTime="372.397636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:04:01.26240813 +0000 UTC m=+2130.943141188" watchObservedRunningTime="2026-07-22 03:04:01.264389459 +0000 UTC m=+2130.945122543"
Jul 22 03:04:07.247215 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:07.247184    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:04:08.080401 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.080365    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"]
Jul 22 03:04:08.083808 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.083792    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.086922 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.086897    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-caf067-serving-cert\""
Jul 22 03:04:08.111205 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.111169    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxbj\" (UniqueName: \"kubernetes.io/projected/de4a865b-3b1d-42f9-a402-0c5581fe9b26-kube-api-access-gmxbj\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.111387 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.111210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4a865b-3b1d-42f9-a402-0c5581fe9b26-openshift-service-ca\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.111387 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.111236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.190695 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.190651    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"]
Jul 22 03:04:08.211923 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.211890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxbj\" (UniqueName: \"kubernetes.io/projected/de4a865b-3b1d-42f9-a402-0c5581fe9b26-kube-api-access-gmxbj\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.212131 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.211935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4a865b-3b1d-42f9-a402-0c5581fe9b26-openshift-service-ca\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.212131 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.211960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.212131 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:04:08.212096    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-serving-cert: secret "maas-api-e2e-inf-b-caf067-serving-cert" not found
Jul 22 03:04:08.212270 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:04:08.212182    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls podName:de4a865b-3b1d-42f9-a402-0c5581fe9b26 nodeName:}" failed. No retries permitted until 2026-07-22 03:04:08.712145649 +0000 UTC m=+2138.392878674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls") pod "maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm" (UID: "de4a865b-3b1d-42f9-a402-0c5581fe9b26") : secret "maas-api-e2e-inf-b-caf067-serving-cert" not found
Jul 22 03:04:08.212735 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.212704    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4a865b-3b1d-42f9-a402-0c5581fe9b26-openshift-service-ca\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.224646 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.224609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxbj\" (UniqueName: \"kubernetes.io/projected/de4a865b-3b1d-42f9-a402-0c5581fe9b26-kube-api-access-gmxbj\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.714966 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.714929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.717549 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.717517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls\") pod \"maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:08.994274 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:08.994242    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:09.141821 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:09.141788    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"]
Jul 22 03:04:09.145110 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:04:09.145074    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4a865b_3b1d_42f9_a402_0c5581fe9b26.slice/crio-c83368cb997245fafba20d0021ccd560765755133de4004f48e477da8519d659 WatchSource:0}: Error finding container c83368cb997245fafba20d0021ccd560765755133de4004f48e477da8519d659: Status 404 returned error can't find the container with id c83368cb997245fafba20d0021ccd560765755133de4004f48e477da8519d659
Jul 22 03:04:09.275289 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:09.275202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm" event={"ID":"de4a865b-3b1d-42f9-a402-0c5581fe9b26","Type":"ContainerStarted","Data":"c83368cb997245fafba20d0021ccd560765755133de4004f48e477da8519d659"}
Jul 22 03:04:10.281492 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:10.281449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm" event={"ID":"de4a865b-3b1d-42f9-a402-0c5581fe9b26","Type":"ContainerStarted","Data":"739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f"}
Jul 22 03:04:10.281887 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:10.281568    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:10.304472 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:10.304417    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm" podStartSLOduration=1.942085635 podStartE2EDuration="2.304396585s" totalImagesPullingTime="362.31095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:04:10.303325864 +0000 UTC m=+2139.984058913" watchObservedRunningTime="2026-07-22 03:04:10.304396585 +0000 UTC m=+2139.985129632"
Jul 22 03:04:16.291993 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:16.291967    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:04:23.519749 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.519667    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"]
Jul 22 03:04:23.523195 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.523177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.525926 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.525901    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-317968\"/\"test-model-317968-kserve-self-signed-certs\""
Jul 22 03:04:23.526829 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.526802    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-317968\"/\"kube-root-ca.crt\""
Jul 22 03:04:23.526962 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.526869    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-317968\"/\"default-dockercfg-tt7sz\""
Jul 22 03:04:23.527059 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.526977    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-317968\"/\"openshift-service-ca.crt\""
Jul 22 03:04:23.532644 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.532622    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"]
Jul 22 03:04:23.647248 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.647209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-model-cache\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.647248 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.647254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tls-certs\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.647486 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.647288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zj5\" (UniqueName: \"kubernetes.io/projected/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-kube-api-access-26zj5\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.647486 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.647323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-dshm\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.647486 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.647341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-home\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.647486 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.647416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tmp-dir\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748223 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748186    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-dshm\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748223 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-home\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tmp-dir\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748316    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-model-cache\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748339    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tls-certs\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748400    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zj5\" (UniqueName: \"kubernetes.io/projected/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-kube-api-access-26zj5\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748682 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748657    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-home\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748746 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tmp-dir\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.748802 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.748765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-model-cache\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.750755 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.750723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-dshm\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.750861 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.750823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tls-certs\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.757594 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.757573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zj5\" (UniqueName: \"kubernetes.io/projected/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-kube-api-access-26zj5\") pod \"test-model-317968-kserve-6c5c867df-44cj8\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") " pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.833927 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.833847    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:23.971602 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:23.971575    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"]
Jul 22 03:04:23.974234 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:04:23.974203    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45fddbcc_e2e6_48a8_acc1_4207a9b2e969.slice/crio-9babf077902ecbe7f3b23fd215a08846640ea5d88354adf1248b73a451db48d7 WatchSource:0}: Error finding container 9babf077902ecbe7f3b23fd215a08846640ea5d88354adf1248b73a451db48d7: Status 404 returned error can't find the container with id 9babf077902ecbe7f3b23fd215a08846640ea5d88354adf1248b73a451db48d7
Jul 22 03:04:24.346758 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:24.346722    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8" event={"ID":"45fddbcc-e2e6-48a8-acc1-4207a9b2e969","Type":"ContainerStarted","Data":"6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9"}
Jul 22 03:04:24.346934 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:24.346771    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8" event={"ID":"45fddbcc-e2e6-48a8-acc1-4207a9b2e969","Type":"ContainerStarted","Data":"9babf077902ecbe7f3b23fd215a08846640ea5d88354adf1248b73a451db48d7"}
Jul 22 03:04:24.387399 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:24.387348    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8" podStartSLOduration=1.387332592 podStartE2EDuration="1.387332592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:04:24.384972661 +0000 UTC m=+2154.065705704" watchObservedRunningTime="2026-07-22 03:04:24.387332592 +0000 UTC m=+2154.068065638"
Jul 22 03:04:33.834441 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:33.834405    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:33.834441 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:33.834451    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:33.846928 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:33.846902    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:04:34.403385 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:04:34.403351    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:05:37.462520 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:37.462482    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"]
Jul 22 03:05:49.311547 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.311515    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"]
Jul 22 03:05:49.312072 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.311703    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v" podUID="6205d85e-ba11-40c3-94db-5dab7408af73" containerName="maas-api" containerID="cri-o://a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd" gracePeriod=30
Jul 22 03:05:49.565966 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.565906    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:05:49.635813 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.635769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6205d85e-ba11-40c3-94db-5dab7408af73-openshift-service-ca\") pod \"6205d85e-ba11-40c3-94db-5dab7408af73\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") "
Jul 22 03:05:49.636004 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.635914    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6205d85e-ba11-40c3-94db-5dab7408af73-maas-api-tls\") pod \"6205d85e-ba11-40c3-94db-5dab7408af73\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") "
Jul 22 03:05:49.636004 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.635954    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kznw6\" (UniqueName: \"kubernetes.io/projected/6205d85e-ba11-40c3-94db-5dab7408af73-kube-api-access-kznw6\") pod \"6205d85e-ba11-40c3-94db-5dab7408af73\" (UID: \"6205d85e-ba11-40c3-94db-5dab7408af73\") "
Jul 22 03:05:49.636237 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.636211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6205d85e-ba11-40c3-94db-5dab7408af73-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6205d85e-ba11-40c3-94db-5dab7408af73" (UID: "6205d85e-ba11-40c3-94db-5dab7408af73"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:05:49.638281 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.638249    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6205d85e-ba11-40c3-94db-5dab7408af73-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6205d85e-ba11-40c3-94db-5dab7408af73" (UID: "6205d85e-ba11-40c3-94db-5dab7408af73"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:05:49.638381 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.638293    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6205d85e-ba11-40c3-94db-5dab7408af73-kube-api-access-kznw6" (OuterVolumeSpecName: "kube-api-access-kznw6") pod "6205d85e-ba11-40c3-94db-5dab7408af73" (UID: "6205d85e-ba11-40c3-94db-5dab7408af73"). InnerVolumeSpecName "kube-api-access-kznw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:05:49.732272 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.732236    2583 generic.go:356] "Generic (PLEG): container finished" podID="6205d85e-ba11-40c3-94db-5dab7408af73" containerID="a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd" exitCode=0
Jul 22 03:05:49.732462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.732296    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"
Jul 22 03:05:49.732462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.732303    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v" event={"ID":"6205d85e-ba11-40c3-94db-5dab7408af73","Type":"ContainerDied","Data":"a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd"}
Jul 22 03:05:49.732462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.732352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v" event={"ID":"6205d85e-ba11-40c3-94db-5dab7408af73","Type":"ContainerDied","Data":"daa06e051840a25493f1d91f13302e1d1ffdf343819bdb88db19aa03830ba63e"}
Jul 22 03:05:49.732462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.732375    2583 scope.go:117] "RemoveContainer" containerID="a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd"
Jul 22 03:05:49.736592 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.736569    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6205d85e-ba11-40c3-94db-5dab7408af73-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:49.736592 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.736595    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kznw6\" (UniqueName: \"kubernetes.io/projected/6205d85e-ba11-40c3-94db-5dab7408af73-kube-api-access-kznw6\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:49.736727 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.736604    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6205d85e-ba11-40c3-94db-5dab7408af73-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:49.741854 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.741838    2583 scope.go:117] "RemoveContainer" containerID="a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd"
Jul 22 03:05:49.742119 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:05:49.742099    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd\": container with ID starting with a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd not found: ID does not exist" containerID="a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd"
Jul 22 03:05:49.742188 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.742130    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd"} err="failed to get container status \"a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd\": rpc error: code = NotFound desc = could not find container \"a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd\": container with ID starting with a077813acc4d478ec9acba11dbe7ae87a231d5c7f9d59295256651c7abb32efd not found: ID does not exist"
Jul 22 03:05:49.760119 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.760088    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"]
Jul 22 03:05:49.764743 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:49.764718    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-317968-64c869fd79-w692v"]
Jul 22 03:05:50.869438 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:50.869357    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6205d85e-ba11-40c3-94db-5dab7408af73" path="/var/lib/kubelet/pods/6205d85e-ba11-40c3-94db-5dab7408af73/volumes"
Jul 22 03:05:52.482742 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.482698    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8" podUID="45fddbcc-e2e6-48a8-acc1-4207a9b2e969" containerName="main" containerID="cri-o://6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9" gracePeriod=45
Jul 22 03:05:52.730198 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.730172    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:05:52.747389 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.747313    2583 generic.go:356] "Generic (PLEG): container finished" podID="45fddbcc-e2e6-48a8-acc1-4207a9b2e969" containerID="6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9" exitCode=0
Jul 22 03:05:52.747389 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.747359    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"
Jul 22 03:05:52.747563 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.747398    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8" event={"ID":"45fddbcc-e2e6-48a8-acc1-4207a9b2e969","Type":"ContainerDied","Data":"6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9"}
Jul 22 03:05:52.747563 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.747439    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8" event={"ID":"45fddbcc-e2e6-48a8-acc1-4207a9b2e969","Type":"ContainerDied","Data":"9babf077902ecbe7f3b23fd215a08846640ea5d88354adf1248b73a451db48d7"}
Jul 22 03:05:52.747563 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.747463    2583 scope.go:117] "RemoveContainer" containerID="6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9"
Jul 22 03:05:52.756844 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.756821    2583 scope.go:117] "RemoveContainer" containerID="6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9"
Jul 22 03:05:52.757256 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:05:52.757227    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9\": container with ID starting with 6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9 not found: ID does not exist" containerID="6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9"
Jul 22 03:05:52.757395 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.757265    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9"} err="failed to get container status \"6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9\": rpc error: code = NotFound desc = could not find container \"6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9\": container with ID starting with 6b609f4724a011f2f2bc32ba6ffeeca5f6ce8a39ae5d1c3196c4a93da75777d9 not found: ID does not exist"
Jul 22 03:05:52.765554 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tls-certs\") pod \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") "
Jul 22 03:05:52.765678 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765579    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26zj5\" (UniqueName: \"kubernetes.io/projected/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-kube-api-access-26zj5\") pod \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") "
Jul 22 03:05:52.765678 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765608    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-home\") pod \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") "
Jul 22 03:05:52.765678 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tmp-dir\") pod \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") "
Jul 22 03:05:52.765836 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765720    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-model-cache\") pod \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") "
Jul 22 03:05:52.765836 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765773    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-dshm\") pod \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\" (UID: \"45fddbcc-e2e6-48a8-acc1-4207a9b2e969\") "
Jul 22 03:05:52.765940 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765899    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-home" (OuterVolumeSpecName: "home") pod "45fddbcc-e2e6-48a8-acc1-4207a9b2e969" (UID: "45fddbcc-e2e6-48a8-acc1-4207a9b2e969"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:05:52.765996 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765950    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45fddbcc-e2e6-48a8-acc1-4207a9b2e969" (UID: "45fddbcc-e2e6-48a8-acc1-4207a9b2e969"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:05:52.765996 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.765965    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-model-cache" (OuterVolumeSpecName: "model-cache") pod "45fddbcc-e2e6-48a8-acc1-4207a9b2e969" (UID: "45fddbcc-e2e6-48a8-acc1-4207a9b2e969"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:05:52.766275 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.766167    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-home\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:52.766275 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.766194    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tmp-dir\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:52.766275 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.766210    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-model-cache\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:52.767937 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.767910    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-kube-api-access-26zj5" (OuterVolumeSpecName: "kube-api-access-26zj5") pod "45fddbcc-e2e6-48a8-acc1-4207a9b2e969" (UID: "45fddbcc-e2e6-48a8-acc1-4207a9b2e969"). InnerVolumeSpecName "kube-api-access-26zj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:05:52.767937 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.767926    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45fddbcc-e2e6-48a8-acc1-4207a9b2e969" (UID: "45fddbcc-e2e6-48a8-acc1-4207a9b2e969"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:05:52.768301 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.768280    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-dshm" (OuterVolumeSpecName: "dshm") pod "45fddbcc-e2e6-48a8-acc1-4207a9b2e969" (UID: "45fddbcc-e2e6-48a8-acc1-4207a9b2e969"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:05:52.866604 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.866574    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-tls-certs\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:52.866604 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.866600    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26zj5\" (UniqueName: \"kubernetes.io/projected/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-kube-api-access-26zj5\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:52.866604 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:52.866609    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45fddbcc-e2e6-48a8-acc1-4207a9b2e969-dshm\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:05:53.067451 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:53.066130    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"]
Jul 22 03:05:53.071085 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:53.071053    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-317968/test-model-317968-kserve-6c5c867df-44cj8"]
Jul 22 03:05:54.869168 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:05:54.869134    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45fddbcc-e2e6-48a8-acc1-4207a9b2e969" path="/var/lib/kubelet/pods/45fddbcc-e2e6-48a8-acc1-4207a9b2e969/volumes"
Jul 22 03:06:01.767953 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.767908    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"]
Jul 22 03:06:01.768461 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.768431    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6205d85e-ba11-40c3-94db-5dab7408af73" containerName="maas-api"
Jul 22 03:06:01.768461 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.768450    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6205d85e-ba11-40c3-94db-5dab7408af73" containerName="maas-api"
Jul 22 03:06:01.768461 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.768461    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45fddbcc-e2e6-48a8-acc1-4207a9b2e969" containerName="main"
Jul 22 03:06:01.768616 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.768469    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="45fddbcc-e2e6-48a8-acc1-4207a9b2e969" containerName="main"
Jul 22 03:06:01.768616 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.768567    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6205d85e-ba11-40c3-94db-5dab7408af73" containerName="maas-api"
Jul 22 03:06:01.768616 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.768578    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="45fddbcc-e2e6-48a8-acc1-4207a9b2e969" containerName="main"
Jul 22 03:06:01.772817 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.772797    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"
Jul 22 03:06:01.775898 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.775881    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\""
Jul 22 03:06:01.788893 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.788868    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"]
Jul 22 03:06:01.849198 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.849160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c602aef-ab66-46f0-96f7-45caf3ad994f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl\" (UID: \"6c602aef-ab66-46f0-96f7-45caf3ad994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"
Jul 22 03:06:01.950017 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.949983    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c602aef-ab66-46f0-96f7-45caf3ad994f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl\" (UID: \"6c602aef-ab66-46f0-96f7-45caf3ad994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"
Jul 22 03:06:01.950621 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:01.950603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c602aef-ab66-46f0-96f7-45caf3ad994f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl\" (UID: \"6c602aef-ab66-46f0-96f7-45caf3ad994f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"
Jul 22 03:06:02.082489 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:02.082396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"
Jul 22 03:06:02.235131 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:02.235097    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"]
Jul 22 03:06:02.238991 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:06:02.238950    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c602aef_ab66_46f0_96f7_45caf3ad994f.slice/crio-598f7d54264abd7ffa49a184527dfdd82298589f83959e49677402ff990161ae WatchSource:0}: Error finding container 598f7d54264abd7ffa49a184527dfdd82298589f83959e49677402ff990161ae: Status 404 returned error can't find the container with id 598f7d54264abd7ffa49a184527dfdd82298589f83959e49677402ff990161ae
Jul 22 03:06:02.793473 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:02.793438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl" event={"ID":"6c602aef-ab66-46f0-96f7-45caf3ad994f","Type":"ContainerStarted","Data":"598f7d54264abd7ffa49a184527dfdd82298589f83959e49677402ff990161ae"}
Jul 22 03:06:05.814307 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:05.814266    2583 generic.go:356] "Generic (PLEG): container finished" podID="6c602aef-ab66-46f0-96f7-45caf3ad994f" containerID="0aacd70116aece25be2bead5a31891b079170774e56751217547ac04ba3b1249" exitCode=0
Jul 22 03:06:05.814760 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:05.814341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl" event={"ID":"6c602aef-ab66-46f0-96f7-45caf3ad994f","Type":"ContainerDied","Data":"0aacd70116aece25be2bead5a31891b079170774e56751217547ac04ba3b1249"}
Jul 22 03:06:06.948196 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:06.948170    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"
Jul 22 03:06:07.102408 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:07.102310    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c602aef-ab66-46f0-96f7-45caf3ad994f-maas-api-service-ca\") pod \"6c602aef-ab66-46f0-96f7-45caf3ad994f\" (UID: \"6c602aef-ab66-46f0-96f7-45caf3ad994f\") "
Jul 22 03:06:07.102698 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:07.102674    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c602aef-ab66-46f0-96f7-45caf3ad994f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c602aef-ab66-46f0-96f7-45caf3ad994f" (UID: "6c602aef-ab66-46f0-96f7-45caf3ad994f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:06:07.203649 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:07.203611    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c602aef-ab66-46f0-96f7-45caf3ad994f-maas-api-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:06:07.824269 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:07.824239    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"
Jul 22 03:06:07.824455 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:07.824237    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl" event={"ID":"6c602aef-ab66-46f0-96f7-45caf3ad994f","Type":"ContainerDied","Data":"598f7d54264abd7ffa49a184527dfdd82298589f83959e49677402ff990161ae"}
Jul 22 03:06:07.824455 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:07.824434    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="598f7d54264abd7ffa49a184527dfdd82298589f83959e49677402ff990161ae"
Jul 22 03:06:12.425006 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.424974    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"]
Jul 22 03:06:12.425459 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.425197    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm" podUID="de4a865b-3b1d-42f9-a402-0c5581fe9b26" containerName="maas-api" containerID="cri-o://739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f" gracePeriod=30
Jul 22 03:06:12.678522 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.678466    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:06:12.846936 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.846903    2583 generic.go:356] "Generic (PLEG): container finished" podID="de4a865b-3b1d-42f9-a402-0c5581fe9b26" containerID="739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f" exitCode=0
Jul 22 03:06:12.847151 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.846960    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"
Jul 22 03:06:12.847151 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.846978    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm" event={"ID":"de4a865b-3b1d-42f9-a402-0c5581fe9b26","Type":"ContainerDied","Data":"739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f"}
Jul 22 03:06:12.847151 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.847020    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm" event={"ID":"de4a865b-3b1d-42f9-a402-0c5581fe9b26","Type":"ContainerDied","Data":"c83368cb997245fafba20d0021ccd560765755133de4004f48e477da8519d659"}
Jul 22 03:06:12.847151 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.847067    2583 scope.go:117] "RemoveContainer" containerID="739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f"
Jul 22 03:06:12.852877 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.852851    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmxbj\" (UniqueName: \"kubernetes.io/projected/de4a865b-3b1d-42f9-a402-0c5581fe9b26-kube-api-access-gmxbj\") pod \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") "
Jul 22 03:06:12.853011 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.852902    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls\") pod \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") "
Jul 22 03:06:12.853011 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.852968    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4a865b-3b1d-42f9-a402-0c5581fe9b26-openshift-service-ca\") pod \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\" (UID: \"de4a865b-3b1d-42f9-a402-0c5581fe9b26\") "
Jul 22 03:06:12.853356 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.853333    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4a865b-3b1d-42f9-a402-0c5581fe9b26-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de4a865b-3b1d-42f9-a402-0c5581fe9b26" (UID: "de4a865b-3b1d-42f9-a402-0c5581fe9b26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:06:12.855206 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.855182    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de4a865b-3b1d-42f9-a402-0c5581fe9b26" (UID: "de4a865b-3b1d-42f9-a402-0c5581fe9b26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:06:12.855377 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.855354    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4a865b-3b1d-42f9-a402-0c5581fe9b26-kube-api-access-gmxbj" (OuterVolumeSpecName: "kube-api-access-gmxbj") pod "de4a865b-3b1d-42f9-a402-0c5581fe9b26" (UID: "de4a865b-3b1d-42f9-a402-0c5581fe9b26"). InnerVolumeSpecName "kube-api-access-gmxbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:06:12.856361 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.856337    2583 scope.go:117] "RemoveContainer" containerID="739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f"
Jul 22 03:06:12.856675 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:06:12.856658    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f\": container with ID starting with 739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f not found: ID does not exist" containerID="739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f"
Jul 22 03:06:12.856741 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.856684    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f"} err="failed to get container status \"739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f\": rpc error: code = NotFound desc = could not find container \"739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f\": container with ID starting with 739a64204c2a70d2565e1e37e415dae7b3cca776dc7f58bc740a971c5812873f not found: ID does not exist"
Jul 22 03:06:12.954247 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.954148    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmxbj\" (UniqueName: \"kubernetes.io/projected/de4a865b-3b1d-42f9-a402-0c5581fe9b26-kube-api-access-gmxbj\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:06:12.954247 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.954184    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de4a865b-3b1d-42f9-a402-0c5581fe9b26-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:06:12.954247 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:12.954194    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4a865b-3b1d-42f9-a402-0c5581fe9b26-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:06:13.188786 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:13.188751    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"]
Jul 22 03:06:13.194188 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:13.194155    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-caf067-5494485d4c-9ckjm"]
Jul 22 03:06:14.868606 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:14.868572    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de4a865b-3b1d-42f9-a402-0c5581fe9b26" path="/var/lib/kubelet/pods/de4a865b-3b1d-42f9-a402-0c5581fe9b26/volumes"
Jul 22 03:06:16.883724 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:16.883696    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"]
Jul 22 03:06:16.891082 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:16.891053    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-caf067-2v5gl"]
Jul 22 03:06:18.868123 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:18.868090    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c602aef-ab66-46f0-96f7-45caf3ad994f" path="/var/lib/kubelet/pods/6c602aef-ab66-46f0-96f7-45caf3ad994f/volumes"
Jul 22 03:06:28.597226 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.597195    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"]
Jul 22 03:06:28.599671 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.597524    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c602aef-ab66-46f0-96f7-45caf3ad994f" containerName="revoke-keys"
Jul 22 03:06:28.599671 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.597535    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c602aef-ab66-46f0-96f7-45caf3ad994f" containerName="revoke-keys"
Jul 22 03:06:28.599671 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.597554    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de4a865b-3b1d-42f9-a402-0c5581fe9b26" containerName="maas-api"
Jul 22 03:06:28.599671 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.597560    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="de4a865b-3b1d-42f9-a402-0c5581fe9b26" containerName="maas-api"
Jul 22 03:06:28.599671 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.597617    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c602aef-ab66-46f0-96f7-45caf3ad994f" containerName="revoke-keys"
Jul 22 03:06:28.599671 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.597627    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="de4a865b-3b1d-42f9-a402-0c5581fe9b26" containerName="maas-api"
Jul 22 03:06:28.600714 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.600684    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.612371 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.612352    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e35e3d-serving-cert\""
Jul 22 03:06:28.647739 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.647705    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"]
Jul 22 03:06:28.692191 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.692154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mcq\" (UniqueName: \"kubernetes.io/projected/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-kube-api-access-54mcq\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.692369 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.692208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-maas-api-tls\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.692369 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.692283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-openshift-service-ca\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.793686 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.793644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mcq\" (UniqueName: \"kubernetes.io/projected/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-kube-api-access-54mcq\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.793899 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.793699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-maas-api-tls\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.793899 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.793742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-openshift-service-ca\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.794458 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.794433    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-openshift-service-ca\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.796425 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.796400    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-maas-api-tls\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.811122 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.811089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mcq\" (UniqueName: \"kubernetes.io/projected/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-kube-api-access-54mcq\") pod \"maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:28.911226 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:28.911134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:29.070828 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:29.070793    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"]
Jul 22 03:06:29.077988 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:06:29.077957    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9526db3f_1ab4_4dec_a3aa_7d3ad9e2ddbf.slice/crio-4057e51b693d646b8a74695c8f6d37bdb04cb5cf641db385b82ddd188c385450 WatchSource:0}: Error finding container 4057e51b693d646b8a74695c8f6d37bdb04cb5cf641db385b82ddd188c385450: Status 404 returned error can't find the container with id 4057e51b693d646b8a74695c8f6d37bdb04cb5cf641db385b82ddd188c385450
Jul 22 03:06:29.079624 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:29.079606    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 03:06:29.919313 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:29.919281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c" event={"ID":"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf","Type":"ContainerStarted","Data":"4057e51b693d646b8a74695c8f6d37bdb04cb5cf641db385b82ddd188c385450"}
Jul 22 03:06:30.926052 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:30.925998    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c" event={"ID":"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf","Type":"ContainerStarted","Data":"b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff"}
Jul 22 03:06:30.926495 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:30.926132    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:30.964360 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:30.964311    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c" podStartSLOduration=2.039584902 podStartE2EDuration="2.964294798s" totalImagesPullingTime="924.709896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:06:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:06:30.963525914 +0000 UTC m=+2280.644258961" watchObservedRunningTime="2026-07-22 03:06:30.964294798 +0000 UTC m=+2280.645027844"
Jul 22 03:06:36.907282 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:36.907244    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"]
Jul 22 03:06:36.916305 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:36.916284    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:36.919251 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:36.919223    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f3887a-serving-cert\""
Jul 22 03:06:36.925042 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:36.925000    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"]
Jul 22 03:06:36.935701 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:36.935672    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:06:36.964958 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:36.964915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:36.965181 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:36.965018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f0e42-7f06-4807-9496-658f20204431-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:36.965181 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:36.965120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vcv8\" (UniqueName: \"kubernetes.io/projected/c03f0e42-7f06-4807-9496-658f20204431-kube-api-access-5vcv8\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.065708 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.065675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vcv8\" (UniqueName: \"kubernetes.io/projected/c03f0e42-7f06-4807-9496-658f20204431-kube-api-access-5vcv8\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.065901 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.065744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.065901 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.065850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f0e42-7f06-4807-9496-658f20204431-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.066023 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:06:37.066005    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-serving-cert: secret "maas-api-e2e-api-b-f3887a-serving-cert" not found
Jul 22 03:06:37.066122 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:06:37.066094    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls podName:c03f0e42-7f06-4807-9496-658f20204431 nodeName:}" failed. No retries permitted until 2026-07-22 03:06:37.56607802 +0000 UTC m=+2287.246811044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls") pod "maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh" (UID: "c03f0e42-7f06-4807-9496-658f20204431") : secret "maas-api-e2e-api-b-f3887a-serving-cert" not found
Jul 22 03:06:37.066528 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.066509    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f0e42-7f06-4807-9496-658f20204431-openshift-service-ca\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.099440 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.099403    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vcv8\" (UniqueName: \"kubernetes.io/projected/c03f0e42-7f06-4807-9496-658f20204431-kube-api-access-5vcv8\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.570466 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.570429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.573022 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.573001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls\") pod \"maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.827735 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.827642    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:37.970050 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:37.970005    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"]
Jul 22 03:06:37.973113 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:06:37.973078    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03f0e42_7f06_4807_9496_658f20204431.slice/crio-26168ba84afeea8a2b16866a78fa14be10277ee45bfc5e9a0fa2622f79d66bbe WatchSource:0}: Error finding container 26168ba84afeea8a2b16866a78fa14be10277ee45bfc5e9a0fa2622f79d66bbe: Status 404 returned error can't find the container with id 26168ba84afeea8a2b16866a78fa14be10277ee45bfc5e9a0fa2622f79d66bbe
Jul 22 03:06:38.969629 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:38.969581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh" event={"ID":"c03f0e42-7f06-4807-9496-658f20204431","Type":"ContainerStarted","Data":"f31ed92c0c0cdf87c6255226873aa900b8f27407090f4e93fdda392f8de33440"}
Jul 22 03:06:38.969815 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:38.969655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh" event={"ID":"c03f0e42-7f06-4807-9496-658f20204431","Type":"ContainerStarted","Data":"26168ba84afeea8a2b16866a78fa14be10277ee45bfc5e9a0fa2622f79d66bbe"}
Jul 22 03:06:38.969815 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:38.969680    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:39.004579 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:39.004525    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh" podStartSLOduration=2.6576050110000002 podStartE2EDuration="3.004506231s" totalImagesPullingTime="346.90122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:06:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:06:39.002765351 +0000 UTC m=+2288.683498396" watchObservedRunningTime="2026-07-22 03:06:39.004506231 +0000 UTC m=+2288.685239279"
Jul 22 03:06:44.980400 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:44.980370    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:06:49.552093 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:49.552051    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"]
Jul 22 03:06:49.556569 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:49.556541    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"
Jul 22 03:06:49.559926 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:49.559899    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\""
Jul 22 03:06:49.578813 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:49.578784    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"]
Jul 22 03:06:49.678288 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:49.678250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96ff546-e412-4b12-ab70-ee7759c89dd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln\" (UID: \"c96ff546-e412-4b12-ab70-ee7759c89dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"
Jul 22 03:06:49.779174 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:49.779140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96ff546-e412-4b12-ab70-ee7759c89dd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln\" (UID: \"c96ff546-e412-4b12-ab70-ee7759c89dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"
Jul 22 03:06:49.779799 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:49.779778    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96ff546-e412-4b12-ab70-ee7759c89dd4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln\" (UID: \"c96ff546-e412-4b12-ab70-ee7759c89dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"
Jul 22 03:06:49.866857 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:49.866766    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"
Jul 22 03:06:50.002580 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:50.002547    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"]
Jul 22 03:06:50.005677 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:06:50.005651    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96ff546_e412_4b12_ab70_ee7759c89dd4.slice/crio-e05cb7851f27a26789688c1e75eacf3fce133e4dbfece39374a2c7babb9158a5 WatchSource:0}: Error finding container e05cb7851f27a26789688c1e75eacf3fce133e4dbfece39374a2c7babb9158a5: Status 404 returned error can't find the container with id e05cb7851f27a26789688c1e75eacf3fce133e4dbfece39374a2c7babb9158a5
Jul 22 03:06:50.018855 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:50.018822    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln" event={"ID":"c96ff546-e412-4b12-ab70-ee7759c89dd4","Type":"ContainerStarted","Data":"e05cb7851f27a26789688c1e75eacf3fce133e4dbfece39374a2c7babb9158a5"}
Jul 22 03:06:51.025326 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:51.025296    2583 generic.go:356] "Generic (PLEG): container finished" podID="c96ff546-e412-4b12-ab70-ee7759c89dd4" containerID="725750f4d6b34acdb96c9b00e3562893841b2131dfa6cca29d59c660adc0d00d" exitCode=0
Jul 22 03:06:51.025747 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:51.025401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln" event={"ID":"c96ff546-e412-4b12-ab70-ee7759c89dd4","Type":"ContainerDied","Data":"725750f4d6b34acdb96c9b00e3562893841b2131dfa6cca29d59c660adc0d00d"}
Jul 22 03:06:52.159425 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:52.159402    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"
Jul 22 03:06:52.302500 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:52.302401    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96ff546-e412-4b12-ab70-ee7759c89dd4-maas-api-service-ca\") pod \"c96ff546-e412-4b12-ab70-ee7759c89dd4\" (UID: \"c96ff546-e412-4b12-ab70-ee7759c89dd4\") "
Jul 22 03:06:52.302826 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:52.302794    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96ff546-e412-4b12-ab70-ee7759c89dd4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c96ff546-e412-4b12-ab70-ee7759c89dd4" (UID: "c96ff546-e412-4b12-ab70-ee7759c89dd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:06:52.403845 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:52.403810    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96ff546-e412-4b12-ab70-ee7759c89dd4-maas-api-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:06:53.035479 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:53.035443    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln" event={"ID":"c96ff546-e412-4b12-ab70-ee7759c89dd4","Type":"ContainerDied","Data":"e05cb7851f27a26789688c1e75eacf3fce133e4dbfece39374a2c7babb9158a5"}
Jul 22 03:06:53.035479 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:53.035480    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"
Jul 22 03:06:53.035725 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:06:53.035490    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e05cb7851f27a26789688c1e75eacf3fce133e4dbfece39374a2c7babb9158a5"
Jul 22 03:07:05.573783 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.573750    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"]
Jul 22 03:07:05.574328 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.573963    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c" podUID="9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" containerName="maas-api" containerID="cri-o://b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff" gracePeriod=30
Jul 22 03:07:05.823433 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.823409    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:07:05.923312 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.923234    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54mcq\" (UniqueName: \"kubernetes.io/projected/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-kube-api-access-54mcq\") pod \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") "
Jul 22 03:07:05.923312 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.923295    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-openshift-service-ca\") pod \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") "
Jul 22 03:07:05.923497 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.923337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-maas-api-tls\") pod \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\" (UID: \"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf\") "
Jul 22 03:07:05.923728 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.923699    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" (UID: "9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:07:05.925442 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.925418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-kube-api-access-54mcq" (OuterVolumeSpecName: "kube-api-access-54mcq") pod "9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" (UID: "9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf"). InnerVolumeSpecName "kube-api-access-54mcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:05.925564 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:05.925511    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" (UID: "9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:07:06.024381 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.024343    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54mcq\" (UniqueName: \"kubernetes.io/projected/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-kube-api-access-54mcq\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:07:06.024381 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.024374    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:07:06.024381 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.024387    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:07:06.096661 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.096624    2583 generic.go:356] "Generic (PLEG): container finished" podID="9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" containerID="b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff" exitCode=0
Jul 22 03:07:06.096838 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.096681    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"
Jul 22 03:07:06.096838 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.096706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c" event={"ID":"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf","Type":"ContainerDied","Data":"b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff"}
Jul 22 03:07:06.096838 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.096758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c" event={"ID":"9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf","Type":"ContainerDied","Data":"4057e51b693d646b8a74695c8f6d37bdb04cb5cf641db385b82ddd188c385450"}
Jul 22 03:07:06.096838 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.096782    2583 scope.go:117] "RemoveContainer" containerID="b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff"
Jul 22 03:07:06.106073 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.106056    2583 scope.go:117] "RemoveContainer" containerID="b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff"
Jul 22 03:07:06.106321 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:07:06.106306    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff\": container with ID starting with b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff not found: ID does not exist" containerID="b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff"
Jul 22 03:07:06.106364 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.106330    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff"} err="failed to get container status \"b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff\": rpc error: code = NotFound desc = could not find container \"b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff\": container with ID starting with b4063f8d37f8fb86cb4292a7e07bca2baf1e0268c1e0163ef8529029284ddeff not found: ID does not exist"
Jul 22 03:07:06.118793 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.118763    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"]
Jul 22 03:07:06.123172 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.123149    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e35e3d-75cf6989b9-wpt6c"]
Jul 22 03:07:06.868820 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:06.868788    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" path="/var/lib/kubelet/pods/9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf/volumes"
Jul 22 03:07:09.608016 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:09.607979    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"]
Jul 22 03:07:09.610151 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:09.610129    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e35e3d-zgtln"]
Jul 22 03:07:10.869514 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:10.869479    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c96ff546-e412-4b12-ab70-ee7759c89dd4" path="/var/lib/kubelet/pods/c96ff546-e412-4b12-ab70-ee7759c89dd4/volumes"
Jul 22 03:07:16.575634 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.575602    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"]
Jul 22 03:07:16.576060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.575937    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c96ff546-e412-4b12-ab70-ee7759c89dd4" containerName="revoke-keys"
Jul 22 03:07:16.576060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.575948    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c96ff546-e412-4b12-ab70-ee7759c89dd4" containerName="revoke-keys"
Jul 22 03:07:16.576060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.575964    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" containerName="maas-api"
Jul 22 03:07:16.576060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.575969    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" containerName="maas-api"
Jul 22 03:07:16.576060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.576023    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9526db3f-1ab4-4dec-a3aa-7d3ad9e2ddbf" containerName="maas-api"
Jul 22 03:07:16.576060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.576046    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c96ff546-e412-4b12-ab70-ee7759c89dd4" containerName="revoke-keys"
Jul 22 03:07:16.579893 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.579876    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"
Jul 22 03:07:16.583001 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.582976    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\""
Jul 22 03:07:16.586104 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.586069    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"]
Jul 22 03:07:16.616279 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.616242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5267a8-b1f2-445a-8742-271c7a1228cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg\" (UID: \"aa5267a8-b1f2-445a-8742-271c7a1228cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"
Jul 22 03:07:16.716860 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.716823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5267a8-b1f2-445a-8742-271c7a1228cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg\" (UID: \"aa5267a8-b1f2-445a-8742-271c7a1228cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"
Jul 22 03:07:16.717496 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.717477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5267a8-b1f2-445a-8742-271c7a1228cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg\" (UID: \"aa5267a8-b1f2-445a-8742-271c7a1228cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"
Jul 22 03:07:16.891770 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:16.891674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"
Jul 22 03:07:17.024711 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:17.024678    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"]
Jul 22 03:07:17.028382 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:07:17.028355    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5267a8_b1f2_445a_8742_271c7a1228cf.slice/crio-5eaebdbf3a20896f7607c734a45b4fe8ceb2e488e64dd2ea7c38048ae3fef6b9 WatchSource:0}: Error finding container 5eaebdbf3a20896f7607c734a45b4fe8ceb2e488e64dd2ea7c38048ae3fef6b9: Status 404 returned error can't find the container with id 5eaebdbf3a20896f7607c734a45b4fe8ceb2e488e64dd2ea7c38048ae3fef6b9
Jul 22 03:07:17.148754 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:17.148672    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg" event={"ID":"aa5267a8-b1f2-445a-8742-271c7a1228cf","Type":"ContainerStarted","Data":"5eaebdbf3a20896f7607c734a45b4fe8ceb2e488e64dd2ea7c38048ae3fef6b9"}
Jul 22 03:07:18.154835 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:18.154734    2583 generic.go:356] "Generic (PLEG): container finished" podID="aa5267a8-b1f2-445a-8742-271c7a1228cf" containerID="184e2d8008f1bd4a1099791b1787e84e2fe7538a61796a5bf2d2996eec802e1f" exitCode=0
Jul 22 03:07:18.154835 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:18.154788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg" event={"ID":"aa5267a8-b1f2-445a-8742-271c7a1228cf","Type":"ContainerDied","Data":"184e2d8008f1bd4a1099791b1787e84e2fe7538a61796a5bf2d2996eec802e1f"}
Jul 22 03:07:19.295661 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:19.295630    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"
Jul 22 03:07:19.338110 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:19.338078    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5267a8-b1f2-445a-8742-271c7a1228cf-maas-api-service-ca\") pod \"aa5267a8-b1f2-445a-8742-271c7a1228cf\" (UID: \"aa5267a8-b1f2-445a-8742-271c7a1228cf\") "
Jul 22 03:07:19.338424 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:19.338400    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5267a8-b1f2-445a-8742-271c7a1228cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa5267a8-b1f2-445a-8742-271c7a1228cf" (UID: "aa5267a8-b1f2-445a-8742-271c7a1228cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:07:19.438768 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:19.438684    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5267a8-b1f2-445a-8742-271c7a1228cf-maas-api-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:07:20.164740 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:20.164716    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"
Jul 22 03:07:20.164924 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:20.164705    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg" event={"ID":"aa5267a8-b1f2-445a-8742-271c7a1228cf","Type":"ContainerDied","Data":"5eaebdbf3a20896f7607c734a45b4fe8ceb2e488e64dd2ea7c38048ae3fef6b9"}
Jul 22 03:07:20.164924 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:20.164848    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eaebdbf3a20896f7607c734a45b4fe8ceb2e488e64dd2ea7c38048ae3fef6b9"
Jul 22 03:07:32.038869 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.038836    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"]
Jul 22 03:07:32.039290 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.039094    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh" podUID="c03f0e42-7f06-4807-9496-658f20204431" containerName="maas-api" containerID="cri-o://f31ed92c0c0cdf87c6255226873aa900b8f27407090f4e93fdda392f8de33440" gracePeriod=30
Jul 22 03:07:32.227907 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.227859    2583 generic.go:356] "Generic (PLEG): container finished" podID="c03f0e42-7f06-4807-9496-658f20204431" containerID="f31ed92c0c0cdf87c6255226873aa900b8f27407090f4e93fdda392f8de33440" exitCode=0
Jul 22 03:07:32.227907 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.227886    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh" event={"ID":"c03f0e42-7f06-4807-9496-658f20204431","Type":"ContainerDied","Data":"f31ed92c0c0cdf87c6255226873aa900b8f27407090f4e93fdda392f8de33440"}
Jul 22 03:07:32.282435 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.282409    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:07:32.350361 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.350279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f0e42-7f06-4807-9496-658f20204431-openshift-service-ca\") pod \"c03f0e42-7f06-4807-9496-658f20204431\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") "
Jul 22 03:07:32.350361 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.350327    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vcv8\" (UniqueName: \"kubernetes.io/projected/c03f0e42-7f06-4807-9496-658f20204431-kube-api-access-5vcv8\") pod \"c03f0e42-7f06-4807-9496-658f20204431\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") "
Jul 22 03:07:32.350594 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.350402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls\") pod \"c03f0e42-7f06-4807-9496-658f20204431\" (UID: \"c03f0e42-7f06-4807-9496-658f20204431\") "
Jul 22 03:07:32.350649 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.350625    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03f0e42-7f06-4807-9496-658f20204431-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c03f0e42-7f06-4807-9496-658f20204431" (UID: "c03f0e42-7f06-4807-9496-658f20204431"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:07:32.352618 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.352595    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f0e42-7f06-4807-9496-658f20204431-kube-api-access-5vcv8" (OuterVolumeSpecName: "kube-api-access-5vcv8") pod "c03f0e42-7f06-4807-9496-658f20204431" (UID: "c03f0e42-7f06-4807-9496-658f20204431"). InnerVolumeSpecName "kube-api-access-5vcv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:07:32.352710 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.352620    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c03f0e42-7f06-4807-9496-658f20204431" (UID: "c03f0e42-7f06-4807-9496-658f20204431"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:07:32.451930 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.451891    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c03f0e42-7f06-4807-9496-658f20204431-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:07:32.451930 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.451924    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03f0e42-7f06-4807-9496-658f20204431-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:07:32.451930 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:32.451937    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vcv8\" (UniqueName: \"kubernetes.io/projected/c03f0e42-7f06-4807-9496-658f20204431-kube-api-access-5vcv8\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:07:33.233731 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:33.233705    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"
Jul 22 03:07:33.234153 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:33.233700    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh" event={"ID":"c03f0e42-7f06-4807-9496-658f20204431","Type":"ContainerDied","Data":"26168ba84afeea8a2b16866a78fa14be10277ee45bfc5e9a0fa2622f79d66bbe"}
Jul 22 03:07:33.234153 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:33.233833    2583 scope.go:117] "RemoveContainer" containerID="f31ed92c0c0cdf87c6255226873aa900b8f27407090f4e93fdda392f8de33440"
Jul 22 03:07:33.252597 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:33.252569    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"]
Jul 22 03:07:33.257844 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:33.257818    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f3887a-68fc4cc74d-z29kh"]
Jul 22 03:07:34.868362 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:34.868334    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03f0e42-7f06-4807-9496-658f20204431" path="/var/lib/kubelet/pods/c03f0e42-7f06-4807-9496-658f20204431/volumes"
Jul 22 03:07:36.673069 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:36.673019    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"]
Jul 22 03:07:36.675854 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:36.675828    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3887a-g9jvg"]
Jul 22 03:07:36.868781 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:36.868746    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5267a8-b1f2-445a-8742-271c7a1228cf" path="/var/lib/kubelet/pods/aa5267a8-b1f2-445a-8742-271c7a1228cf/volumes"
Jul 22 03:07:46.979819 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.979788    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"]
Jul 22 03:07:46.980334 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.980318    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03f0e42-7f06-4807-9496-658f20204431" containerName="maas-api"
Jul 22 03:07:46.980379 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.980337    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03f0e42-7f06-4807-9496-658f20204431" containerName="maas-api"
Jul 22 03:07:46.980379 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.980356    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5267a8-b1f2-445a-8742-271c7a1228cf" containerName="revoke-keys"
Jul 22 03:07:46.980379 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.980365    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5267a8-b1f2-445a-8742-271c7a1228cf" containerName="revoke-keys"
Jul 22 03:07:46.980472 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.980456    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03f0e42-7f06-4807-9496-658f20204431" containerName="maas-api"
Jul 22 03:07:46.980472 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.980468    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5267a8-b1f2-445a-8742-271c7a1228cf" containerName="revoke-keys"
Jul 22 03:07:46.983907 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.983885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:46.986601 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.986583    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-643135-serving-cert\""
Jul 22 03:07:46.995179 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:46.995153    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"]
Jul 22 03:07:47.077757 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.077727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.077961 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.077786    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9074df5-289c-4035-a283-78c13fdb756c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.077961 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.077907    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j222k\" (UniqueName: \"kubernetes.io/projected/d9074df5-289c-4035-a283-78c13fdb756c-kube-api-access-j222k\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.178944 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.178909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j222k\" (UniqueName: \"kubernetes.io/projected/d9074df5-289c-4035-a283-78c13fdb756c-kube-api-access-j222k\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.179157 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.178986    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.179157 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.179061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9074df5-289c-4035-a283-78c13fdb756c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.179239 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:07:47.179158    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-serving-cert: secret "maas-api-e2e-shared-a-643135-serving-cert" not found
Jul 22 03:07:47.179239 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:07:47.179231    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls podName:d9074df5-289c-4035-a283-78c13fdb756c nodeName:}" failed. No retries permitted until 2026-07-22 03:07:47.679211172 +0000 UTC m=+2357.359944209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls") pod "maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4" (UID: "d9074df5-289c-4035-a283-78c13fdb756c") : secret "maas-api-e2e-shared-a-643135-serving-cert" not found
Jul 22 03:07:47.179652 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.179636    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9074df5-289c-4035-a283-78c13fdb756c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.196705 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.196669    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j222k\" (UniqueName: \"kubernetes.io/projected/d9074df5-289c-4035-a283-78c13fdb756c-kube-api-access-j222k\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.683109 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:47.683076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:47.683293 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:07:47.683238    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-serving-cert: secret "maas-api-e2e-shared-a-643135-serving-cert" not found
Jul 22 03:07:47.683338 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:07:47.683327    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls podName:d9074df5-289c-4035-a283-78c13fdb756c nodeName:}" failed. No retries permitted until 2026-07-22 03:07:48.683310908 +0000 UTC m=+2358.364043932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls") pod "maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4" (UID: "d9074df5-289c-4035-a283-78c13fdb756c") : secret "maas-api-e2e-shared-a-643135-serving-cert" not found
Jul 22 03:07:48.692687 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:48.692638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:48.695269 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:48.695241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls\") pod \"maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:48.795044 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:48.794992    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:48.931216 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:48.931193    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"]
Jul 22 03:07:48.933682 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:07:48.933652    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9074df5_289c_4035_a283_78c13fdb756c.slice/crio-f92823d5b7f8adfa98620011f13e3969bcc1ed3b78cb0808a42d3d8f7716aa1c WatchSource:0}: Error finding container f92823d5b7f8adfa98620011f13e3969bcc1ed3b78cb0808a42d3d8f7716aa1c: Status 404 returned error can't find the container with id f92823d5b7f8adfa98620011f13e3969bcc1ed3b78cb0808a42d3d8f7716aa1c
Jul 22 03:07:49.316712 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:49.316683    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4" event={"ID":"d9074df5-289c-4035-a283-78c13fdb756c","Type":"ContainerStarted","Data":"f92823d5b7f8adfa98620011f13e3969bcc1ed3b78cb0808a42d3d8f7716aa1c"}
Jul 22 03:07:50.323172 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:50.323129    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4" event={"ID":"d9074df5-289c-4035-a283-78c13fdb756c","Type":"ContainerStarted","Data":"ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e"}
Jul 22 03:07:50.323625 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:50.323196    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:50.339244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:50.339195    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4" podStartSLOduration=3.984648616 podStartE2EDuration="4.339181094s" totalImagesPullingTime="354.532478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:07:50.338208429 +0000 UTC m=+2360.018941478" watchObservedRunningTime="2026-07-22 03:07:50.339181094 +0000 UTC m=+2360.019914140"
Jul 22 03:07:55.231021 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.230985    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"]
Jul 22 03:07:55.234537 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.234520    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.237148 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.237131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e6d029-serving-cert\""
Jul 22 03:07:55.250122 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.250093    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"]
Jul 22 03:07:55.351406 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.351377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d734b7ff-3e92-4c19-953f-9839786ceeb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.351613 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.351415    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d734b7ff-3e92-4c19-953f-9839786ceeb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.351613 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.351449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxchh\" (UniqueName: \"kubernetes.io/projected/d734b7ff-3e92-4c19-953f-9839786ceeb7-kube-api-access-pxchh\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.452695 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.452666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d734b7ff-3e92-4c19-953f-9839786ceeb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.452879 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.452705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d734b7ff-3e92-4c19-953f-9839786ceeb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.452879 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.452838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxchh\" (UniqueName: \"kubernetes.io/projected/d734b7ff-3e92-4c19-953f-9839786ceeb7-kube-api-access-pxchh\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.453297 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.453278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d734b7ff-3e92-4c19-953f-9839786ceeb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.455162 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.455140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d734b7ff-3e92-4c19-953f-9839786ceeb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.462045 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.462009    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxchh\" (UniqueName: \"kubernetes.io/projected/d734b7ff-3e92-4c19-953f-9839786ceeb7-kube-api-access-pxchh\") pod \"maas-api-e2e-shared-b-e6d029-7684665ccb-84m58\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.545874 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.545795    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:07:55.691827 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:55.691794    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"]
Jul 22 03:07:55.695131 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:07:55.695097    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd734b7ff_3e92_4c19_953f_9839786ceeb7.slice/crio-742acb20c79b98842bf702f12f458882f7070c4dbf5ccb12048586ce3416d8f3 WatchSource:0}: Error finding container 742acb20c79b98842bf702f12f458882f7070c4dbf5ccb12048586ce3416d8f3: Status 404 returned error can't find the container with id 742acb20c79b98842bf702f12f458882f7070c4dbf5ccb12048586ce3416d8f3
Jul 22 03:07:56.334522 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:56.334446    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:07:56.353273 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:56.353240    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58" event={"ID":"d734b7ff-3e92-4c19-953f-9839786ceeb7","Type":"ContainerStarted","Data":"1b6478c184badeb16831b8932700e4503806bc62c64a5c131b8559fbc467aab1"}
Jul 22 03:07:56.353426 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:56.353288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58" event={"ID":"d734b7ff-3e92-4c19-953f-9839786ceeb7","Type":"ContainerStarted","Data":"742acb20c79b98842bf702f12f458882f7070c4dbf5ccb12048586ce3416d8f3"}
Jul 22 03:07:56.353426 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:07:56.353369    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:08:02.364409 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:02.364376    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:08:02.384313 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:02.384261    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58" podStartSLOduration=7.073483312 podStartE2EDuration="7.38424468s" totalImagesPullingTime="310.761368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:07:56.375845206 +0000 UTC m=+2366.056578253" watchObservedRunningTime="2026-07-22 03:08:02.38424468 +0000 UTC m=+2372.064977726"
Jul 22 03:08:09.216464 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.216426    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"]
Jul 22 03:08:09.219574 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.219558    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.223625 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.223595    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"default-dockercfg-5w4zg\""
Jul 22 03:08:09.223791 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.223674    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"auth-test-model-643135-kserve-self-signed-certs\""
Jul 22 03:08:09.223791 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.223738    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"kube-root-ca.crt\""
Jul 22 03:08:09.225260 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.225242    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"openshift-service-ca.crt\""
Jul 22 03:08:09.234530 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.234506    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"]
Jul 22 03:08:09.276762 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.276728    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tmp-dir\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.276955 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.276770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-dshm\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.276955 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.276788    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59z4\" (UniqueName: \"kubernetes.io/projected/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-kube-api-access-f59z4\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.276955 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.276894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tls-certs\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.276955 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.276939    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-model-cache\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.277186 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.277000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-home\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.377505 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.377465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-home\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.377706 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.377515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tmp-dir\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.377706 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.377545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-dshm\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.377706 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.377584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59z4\" (UniqueName: \"kubernetes.io/projected/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-kube-api-access-f59z4\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.377706 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.377634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tls-certs\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.377706 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.377665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-model-cache\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.377992 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.377945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tmp-dir\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.377992 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.377958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-home\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.378138 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.378122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-model-cache\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.380131 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.380113    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-dshm\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.380256 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.380240    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tls-certs\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.392669 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.392649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59z4\" (UniqueName: \"kubernetes.io/projected/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-kube-api-access-f59z4\") pod \"auth-test-model-643135-kserve-5d97746955-25pxv\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") " pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.529621 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.529587    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:09.664337 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:09.664302    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"]
Jul 22 03:08:09.666527 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:08:09.666497    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09178a37_08fa_4ae2_bd53_aae9bc30f8c0.slice/crio-2c4032282c1232e6173ed9e2377a276ac20796dbda2619af19e9af7573fd6f8d WatchSource:0}: Error finding container 2c4032282c1232e6173ed9e2377a276ac20796dbda2619af19e9af7573fd6f8d: Status 404 returned error can't find the container with id 2c4032282c1232e6173ed9e2377a276ac20796dbda2619af19e9af7573fd6f8d
Jul 22 03:08:10.425004 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:10.424911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv" event={"ID":"09178a37-08fa-4ae2-bd53-aae9bc30f8c0","Type":"ContainerStarted","Data":"5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834"}
Jul 22 03:08:10.425004 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:10.424960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv" event={"ID":"09178a37-08fa-4ae2-bd53-aae9bc30f8c0","Type":"ContainerStarted","Data":"2c4032282c1232e6173ed9e2377a276ac20796dbda2619af19e9af7573fd6f8d"}
Jul 22 03:08:10.444462 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:10.444417    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv" podStartSLOduration=1.444401256 podStartE2EDuration="1.444401256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:08:10.443090686 +0000 UTC m=+2380.123823733" watchObservedRunningTime="2026-07-22 03:08:10.444401256 +0000 UTC m=+2380.125134301"
Jul 22 03:08:19.529835 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:19.529796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:19.530273 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:19.530085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:19.542570 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:19.542548    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:20.485552 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:20.485521    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:08:30.824172 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.824138    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"]
Jul 22 03:08:30.828427 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.828410    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:30.831649 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.831628    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"openshift-service-ca.crt\""
Jul 22 03:08:30.831840 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.831820    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"default-dockercfg-x8dhn\""
Jul 22 03:08:30.832640 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.832620    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"kube-root-ca.crt\""
Jul 22 03:08:30.832749 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.832739    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"auth-test-model-e6d029-kserve-self-signed-certs\""
Jul 22 03:08:30.841609 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.841585    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"]
Jul 22 03:08:30.916433 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.916403    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 03:08:30.918894 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.918872    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 03:08:30.966124 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.966081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad23370d-a599-4d29-a039-f746b03eaa7b-tls-certs\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:30.966281 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.966155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-tmp-dir\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:30.966281 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.966220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-home\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:30.966281 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.966260    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-model-cache\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:30.966407 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.966289    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jl9f\" (UniqueName: \"kubernetes.io/projected/ad23370d-a599-4d29-a039-f746b03eaa7b-kube-api-access-4jl9f\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:30.966407 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:30.966331    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-dshm\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067343 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad23370d-a599-4d29-a039-f746b03eaa7b-tls-certs\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067515 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-tmp-dir\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067515 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-home\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067515 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067454    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-model-cache\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067625 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jl9f\" (UniqueName: \"kubernetes.io/projected/ad23370d-a599-4d29-a039-f746b03eaa7b-kube-api-access-4jl9f\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067668 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067634    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-dshm\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067875 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067857    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-model-cache\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067941 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067913    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-home\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.067941 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.067930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-tmp-dir\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.069938 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.069909    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-dshm\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.070105 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.070088    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad23370d-a599-4d29-a039-f746b03eaa7b-tls-certs\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.078317 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.078256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jl9f\" (UniqueName: \"kubernetes.io/projected/ad23370d-a599-4d29-a039-f746b03eaa7b-kube-api-access-4jl9f\") pod \"auth-test-model-e6d029-kserve-59f6455755-5dtf5\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") " pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.138705 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.138668    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:31.295829 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.295804    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"]
Jul 22 03:08:31.297965 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:08:31.297942    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad23370d_a599_4d29_a039_f746b03eaa7b.slice/crio-5d253bc8175787d7dc06a65820d0d5217286b849b3576d3b193f8c449d61c452 WatchSource:0}: Error finding container 5d253bc8175787d7dc06a65820d0d5217286b849b3576d3b193f8c449d61c452: Status 404 returned error can't find the container with id 5d253bc8175787d7dc06a65820d0d5217286b849b3576d3b193f8c449d61c452
Jul 22 03:08:31.529603 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.529550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5" event={"ID":"ad23370d-a599-4d29-a039-f746b03eaa7b","Type":"ContainerStarted","Data":"bfede545bd68e33566dc057a20f54b6f20a07ec8f6b58b24ae3a84046d517da0"}
Jul 22 03:08:31.529791 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:31.529616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5" event={"ID":"ad23370d-a599-4d29-a039-f746b03eaa7b","Type":"ContainerStarted","Data":"5d253bc8175787d7dc06a65820d0d5217286b849b3576d3b193f8c449d61c452"}
Jul 22 03:08:41.139610 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:41.139574    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:41.140169 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:41.139624    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:41.152182 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:41.152150    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:08:41.174315 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:41.174256    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5" podStartSLOduration=11.174236167 podStartE2EDuration="11.174236167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:08:31.567115246 +0000 UTC m=+2401.247848292" watchObservedRunningTime="2026-07-22 03:08:41.174236167 +0000 UTC m=+2410.854969213"
Jul 22 03:08:41.590239 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:08:41.590208    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:10:40.084478 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:40.084443    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"]
Jul 22 03:10:40.978557 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:40.978521    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"]
Jul 22 03:10:55.104422 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.104372    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv" podUID="09178a37-08fa-4ae2-bd53-aae9bc30f8c0" containerName="main" containerID="cri-o://5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834" gracePeriod=45
Jul 22 03:10:55.347977 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.347948    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:10:55.500157 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f59z4\" (UniqueName: \"kubernetes.io/projected/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-kube-api-access-f59z4\") pod \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") "
Jul 22 03:10:55.500319 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500173    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-dshm\") pod \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") "
Jul 22 03:10:55.500319 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500198    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-model-cache\") pod \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") "
Jul 22 03:10:55.500319 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500307    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tmp-dir\") pod \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") "
Jul 22 03:10:55.500467 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500332    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tls-certs\") pod \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") "
Jul 22 03:10:55.500467 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500354    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-home\") pod \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\" (UID: \"09178a37-08fa-4ae2-bd53-aae9bc30f8c0\") "
Jul 22 03:10:55.500574 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500466    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "09178a37-08fa-4ae2-bd53-aae9bc30f8c0" (UID: "09178a37-08fa-4ae2-bd53-aae9bc30f8c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:10:55.500633 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500586    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09178a37-08fa-4ae2-bd53-aae9bc30f8c0" (UID: "09178a37-08fa-4ae2-bd53-aae9bc30f8c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:10:55.500693 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500675    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-home" (OuterVolumeSpecName: "home") pod "09178a37-08fa-4ae2-bd53-aae9bc30f8c0" (UID: "09178a37-08fa-4ae2-bd53-aae9bc30f8c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:10:55.500785 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500770    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-model-cache\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:55.500785 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500784    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tmp-dir\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:55.500878 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.500792    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-home\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:55.502552 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.502532    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-dshm" (OuterVolumeSpecName: "dshm") pod "09178a37-08fa-4ae2-bd53-aae9bc30f8c0" (UID: "09178a37-08fa-4ae2-bd53-aae9bc30f8c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:10:55.502552 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.502536    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-kube-api-access-f59z4" (OuterVolumeSpecName: "kube-api-access-f59z4") pod "09178a37-08fa-4ae2-bd53-aae9bc30f8c0" (UID: "09178a37-08fa-4ae2-bd53-aae9bc30f8c0"). InnerVolumeSpecName "kube-api-access-f59z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:10:55.502658 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.502607    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09178a37-08fa-4ae2-bd53-aae9bc30f8c0" (UID: "09178a37-08fa-4ae2-bd53-aae9bc30f8c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:10:55.602077 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.602009    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-tls-certs\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:55.602077 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.602065    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59z4\" (UniqueName: \"kubernetes.io/projected/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-kube-api-access-f59z4\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:55.602077 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:55.602075    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09178a37-08fa-4ae2-bd53-aae9bc30f8c0-dshm\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:56.001613 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.001571    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5" podUID="ad23370d-a599-4d29-a039-f746b03eaa7b" containerName="main" containerID="cri-o://bfede545bd68e33566dc057a20f54b6f20a07ec8f6b58b24ae3a84046d517da0" gracePeriod=45
Jul 22 03:10:56.208505 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.208469    2583 generic.go:356] "Generic (PLEG): container finished" podID="09178a37-08fa-4ae2-bd53-aae9bc30f8c0" containerID="5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834" exitCode=0
Jul 22 03:10:56.208936 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.208515    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"
Jul 22 03:10:56.208936 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.208549    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv" event={"ID":"09178a37-08fa-4ae2-bd53-aae9bc30f8c0","Type":"ContainerDied","Data":"5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834"}
Jul 22 03:10:56.208936 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.208595    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv" event={"ID":"09178a37-08fa-4ae2-bd53-aae9bc30f8c0","Type":"ContainerDied","Data":"2c4032282c1232e6173ed9e2377a276ac20796dbda2619af19e9af7573fd6f8d"}
Jul 22 03:10:56.208936 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.208619    2583 scope.go:117] "RemoveContainer" containerID="5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834"
Jul 22 03:10:56.210953 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.210925    2583 generic.go:356] "Generic (PLEG): container finished" podID="ad23370d-a599-4d29-a039-f746b03eaa7b" containerID="bfede545bd68e33566dc057a20f54b6f20a07ec8f6b58b24ae3a84046d517da0" exitCode=0
Jul 22 03:10:56.211125 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.210972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5" event={"ID":"ad23370d-a599-4d29-a039-f746b03eaa7b","Type":"ContainerDied","Data":"bfede545bd68e33566dc057a20f54b6f20a07ec8f6b58b24ae3a84046d517da0"}
Jul 22 03:10:56.218273 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.218252    2583 scope.go:117] "RemoveContainer" containerID="5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834"
Jul 22 03:10:56.218618 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:10:56.218597    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834\": container with ID starting with 5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834 not found: ID does not exist" containerID="5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834"
Jul 22 03:10:56.218671 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.218628    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834"} err="failed to get container status \"5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834\": rpc error: code = NotFound desc = could not find container \"5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834\": container with ID starting with 5cc5d0ba66f854692507da705082f68c6066bc29542eaba50007bc9be1921834 not found: ID does not exist"
Jul 22 03:10:56.233797 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.233772    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"]
Jul 22 03:10:56.242108 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.242082    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-643135/auth-test-model-643135-kserve-5d97746955-25pxv"]
Jul 22 03:10:56.262325 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.262265    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:10:56.408589 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad23370d-a599-4d29-a039-f746b03eaa7b-tls-certs\") pod \"ad23370d-a599-4d29-a039-f746b03eaa7b\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") "
Jul 22 03:10:56.408788 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408620    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-home\") pod \"ad23370d-a599-4d29-a039-f746b03eaa7b\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") "
Jul 22 03:10:56.408788 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408649    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jl9f\" (UniqueName: \"kubernetes.io/projected/ad23370d-a599-4d29-a039-f746b03eaa7b-kube-api-access-4jl9f\") pod \"ad23370d-a599-4d29-a039-f746b03eaa7b\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") "
Jul 22 03:10:56.408788 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408680    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-tmp-dir\") pod \"ad23370d-a599-4d29-a039-f746b03eaa7b\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") "
Jul 22 03:10:56.408788 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408696    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-model-cache\") pod \"ad23370d-a599-4d29-a039-f746b03eaa7b\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") "
Jul 22 03:10:56.408788 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-dshm\") pod \"ad23370d-a599-4d29-a039-f746b03eaa7b\" (UID: \"ad23370d-a599-4d29-a039-f746b03eaa7b\") "
Jul 22 03:10:56.409121 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408894    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-home" (OuterVolumeSpecName: "home") pod "ad23370d-a599-4d29-a039-f746b03eaa7b" (UID: "ad23370d-a599-4d29-a039-f746b03eaa7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:10:56.409121 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408985    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad23370d-a599-4d29-a039-f746b03eaa7b" (UID: "ad23370d-a599-4d29-a039-f746b03eaa7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:10:56.409121 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.408988    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad23370d-a599-4d29-a039-f746b03eaa7b" (UID: "ad23370d-a599-4d29-a039-f746b03eaa7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:10:56.411021 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.410993    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-dshm" (OuterVolumeSpecName: "dshm") pod "ad23370d-a599-4d29-a039-f746b03eaa7b" (UID: "ad23370d-a599-4d29-a039-f746b03eaa7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:10:56.411138 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.411045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad23370d-a599-4d29-a039-f746b03eaa7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad23370d-a599-4d29-a039-f746b03eaa7b" (UID: "ad23370d-a599-4d29-a039-f746b03eaa7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:10:56.411138 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.411024    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad23370d-a599-4d29-a039-f746b03eaa7b-kube-api-access-4jl9f" (OuterVolumeSpecName: "kube-api-access-4jl9f") pod "ad23370d-a599-4d29-a039-f746b03eaa7b" (UID: "ad23370d-a599-4d29-a039-f746b03eaa7b"). InnerVolumeSpecName "kube-api-access-4jl9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:10:56.510258 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.510219    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad23370d-a599-4d29-a039-f746b03eaa7b-tls-certs\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:56.510258 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.510248    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-home\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:56.510258 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.510257    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jl9f\" (UniqueName: \"kubernetes.io/projected/ad23370d-a599-4d29-a039-f746b03eaa7b-kube-api-access-4jl9f\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:56.510258 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.510267    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-tmp-dir\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:56.510533 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.510275    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-model-cache\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:56.510533 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.510283    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad23370d-a599-4d29-a039-f746b03eaa7b-dshm\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:10:56.869836 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:56.869805    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09178a37-08fa-4ae2-bd53-aae9bc30f8c0" path="/var/lib/kubelet/pods/09178a37-08fa-4ae2-bd53-aae9bc30f8c0/volumes"
Jul 22 03:10:57.218060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:57.217957    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"
Jul 22 03:10:57.218060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:57.217999    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5" event={"ID":"ad23370d-a599-4d29-a039-f746b03eaa7b","Type":"ContainerDied","Data":"5d253bc8175787d7dc06a65820d0d5217286b849b3576d3b193f8c449d61c452"}
Jul 22 03:10:57.218649 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:57.218065    2583 scope.go:117] "RemoveContainer" containerID="bfede545bd68e33566dc057a20f54b6f20a07ec8f6b58b24ae3a84046d517da0"
Jul 22 03:10:57.236772 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:57.236746    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"]
Jul 22 03:10:57.243380 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:57.243353    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/auth-test-model-e6d029-kserve-59f6455755-5dtf5"]
Jul 22 03:10:58.869448 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:10:58.869404    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad23370d-a599-4d29-a039-f746b03eaa7b" path="/var/lib/kubelet/pods/ad23370d-a599-4d29-a039-f746b03eaa7b/volumes"
Jul 22 03:11:01.416593 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.416557    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"]
Jul 22 03:11:01.417124 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.417053    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad23370d-a599-4d29-a039-f746b03eaa7b" containerName="main"
Jul 22 03:11:01.417124 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.417071    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad23370d-a599-4d29-a039-f746b03eaa7b" containerName="main"
Jul 22 03:11:01.417124 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.417098    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09178a37-08fa-4ae2-bd53-aae9bc30f8c0" containerName="main"
Jul 22 03:11:01.417124 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.417107    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="09178a37-08fa-4ae2-bd53-aae9bc30f8c0" containerName="main"
Jul 22 03:11:01.417438 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.417207    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="09178a37-08fa-4ae2-bd53-aae9bc30f8c0" containerName="main"
Jul 22 03:11:01.417438 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.417221    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad23370d-a599-4d29-a039-f746b03eaa7b" containerName="main"
Jul 22 03:11:01.421795 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.421771    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.424863 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.424838    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"default-dockercfg-x8dhn\""
Jul 22 03:11:01.425193 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.425179    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"openshift-service-ca.crt\""
Jul 22 03:11:01.425892 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.425873    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"kube-root-ca.crt\""
Jul 22 03:11:01.429638 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.429289    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-e6d029\"/\"sub-test-model-e6d029-kserve-self-signed-certs\""
Jul 22 03:11:01.434882 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.434853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"]
Jul 22 03:11:01.453567 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.453540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-dshm\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.453763 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.453594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tmp-dir\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.453763 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.453642    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tls-certs\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.453763 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.453677    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-home\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.453763 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.453727    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7kmb\" (UniqueName: \"kubernetes.io/projected/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-kube-api-access-m7kmb\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.453992 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.453884    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-model-cache\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.554821 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.554786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-dshm\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.554821 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.554819    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tmp-dir\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.555114 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.554846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tls-certs\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.555114 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.554868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-home\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.555114 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.555006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7kmb\" (UniqueName: \"kubernetes.io/projected/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-kube-api-access-m7kmb\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.555287 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.555149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-model-cache\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.555287 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.555253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-home\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.555382 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.555297    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tmp-dir\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.555467 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.555448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-model-cache\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.557412 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.557385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-dshm\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.557605 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.557589    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tls-certs\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.567293 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.567269    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7kmb\" (UniqueName: \"kubernetes.io/projected/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-kube-api-access-m7kmb\") pod \"sub-test-model-e6d029-kserve-796cf85499-wgkgs\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") " pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.732297 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.732260    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:01.882153 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:01.882122    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"]
Jul 22 03:11:01.885775 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:11:01.885750    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7d8e4c2_4057_4b20_8874_20873ed0ff7f.slice/crio-da7f38deea703b78bfb1158fa1499145060cbdcbb9af929e2c22982e7b4db37b WatchSource:0}: Error finding container da7f38deea703b78bfb1158fa1499145060cbdcbb9af929e2c22982e7b4db37b: Status 404 returned error can't find the container with id da7f38deea703b78bfb1158fa1499145060cbdcbb9af929e2c22982e7b4db37b
Jul 22 03:11:02.245445 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:02.245412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs" event={"ID":"c7d8e4c2-4057-4b20-8874-20873ed0ff7f","Type":"ContainerStarted","Data":"b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869"}
Jul 22 03:11:02.245445 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:02.245450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs" event={"ID":"c7d8e4c2-4057-4b20-8874-20873ed0ff7f","Type":"ContainerStarted","Data":"da7f38deea703b78bfb1158fa1499145060cbdcbb9af929e2c22982e7b4db37b"}
Jul 22 03:11:02.276223 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:02.276176    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs" podStartSLOduration=1.27615587 podStartE2EDuration="1.27615587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:11:02.274615152 +0000 UTC m=+2551.955348199" watchObservedRunningTime="2026-07-22 03:11:02.27615587 +0000 UTC m=+2551.956888915"
Jul 22 03:11:11.732992 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:11.732938    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:11.732992 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:11.733002    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:11.745642 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:11.745615    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:12.303871 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:12.303843    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:11:45.604695 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:45.604657    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"]
Jul 22 03:11:55.486855 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.486761    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"]
Jul 22 03:11:55.490387 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.490366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.493724 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.493701    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"default-dockercfg-5w4zg\""
Jul 22 03:11:55.493724 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.493717    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"kube-root-ca.crt\""
Jul 22 03:11:55.493935 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.493751    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"openshift-service-ca.crt\""
Jul 22 03:11:55.493935 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.493751    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-643135\"/\"rate-test-model-643135-kserve-self-signed-certs\""
Jul 22 03:11:55.501639 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.501612    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"]
Jul 22 03:11:55.520138 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.520096    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-tmp-dir\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.520345 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.520191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-model-cache\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.520345 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.520234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dcr\" (UniqueName: \"kubernetes.io/projected/746176e7-6341-4b6c-8edb-8b072e0445a3-kube-api-access-w4dcr\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.520345 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.520302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746176e7-6341-4b6c-8edb-8b072e0445a3-tls-certs\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.520499 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.520344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-home\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.520499 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.520374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-dshm\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621328 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-tmp-dir\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621540 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-model-cache\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621540 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dcr\" (UniqueName: \"kubernetes.io/projected/746176e7-6341-4b6c-8edb-8b072e0445a3-kube-api-access-w4dcr\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621540 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746176e7-6341-4b6c-8edb-8b072e0445a3-tls-certs\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621540 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-home\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621540 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-dshm\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621818 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-model-cache\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621818 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-tmp-dir\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.621907 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.621855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-home\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.623782 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.623765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-dshm\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.624058 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.624019    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746176e7-6341-4b6c-8edb-8b072e0445a3-tls-certs\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.630520 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.630496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dcr\" (UniqueName: \"kubernetes.io/projected/746176e7-6341-4b6c-8edb-8b072e0445a3-kube-api-access-w4dcr\") pod \"rate-test-model-643135-kserve-77b688d779-ghk8r\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") " pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.801355 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.801247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:11:55.935560 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:55.935535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"]
Jul 22 03:11:55.937918 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:11:55.937892    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod746176e7_6341_4b6c_8edb_8b072e0445a3.slice/crio-5877ad7b6dce477cf082798462e3ad0612fdf137adb036d2443ead0d5e1ed68e WatchSource:0}: Error finding container 5877ad7b6dce477cf082798462e3ad0612fdf137adb036d2443ead0d5e1ed68e: Status 404 returned error can't find the container with id 5877ad7b6dce477cf082798462e3ad0612fdf137adb036d2443ead0d5e1ed68e
Jul 22 03:11:56.512609 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:56.512576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r" event={"ID":"746176e7-6341-4b6c-8edb-8b072e0445a3","Type":"ContainerStarted","Data":"a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a"}
Jul 22 03:11:56.512609 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:56.512615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r" event={"ID":"746176e7-6341-4b6c-8edb-8b072e0445a3","Type":"ContainerStarted","Data":"5877ad7b6dce477cf082798462e3ad0612fdf137adb036d2443ead0d5e1ed68e"}
Jul 22 03:11:56.530688 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:11:56.530623    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r" podStartSLOduration=1.530600433 podStartE2EDuration="1.530600433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:11:56.52851489 +0000 UTC m=+2606.209247930" watchObservedRunningTime="2026-07-22 03:11:56.530600433 +0000 UTC m=+2606.211333480"
Jul 22 03:12:00.626704 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.626643    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs" podUID="c7d8e4c2-4057-4b20-8874-20873ed0ff7f" containerName="main" containerID="cri-o://b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869" gracePeriod=45
Jul 22 03:12:00.883421 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.883360    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:12:00.964730 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.964693    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-home\") pod \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") "
Jul 22 03:12:00.964908 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.964771    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-dshm\") pod \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") "
Jul 22 03:12:00.964908 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.964798    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tls-certs\") pod \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") "
Jul 22 03:12:00.964908 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.964821    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7kmb\" (UniqueName: \"kubernetes.io/projected/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-kube-api-access-m7kmb\") pod \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") "
Jul 22 03:12:00.964908 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.964854    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tmp-dir\") pod \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") "
Jul 22 03:12:00.964908 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.964880    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-model-cache\") pod \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\" (UID: \"c7d8e4c2-4057-4b20-8874-20873ed0ff7f\") "
Jul 22 03:12:00.965225 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.965055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-home" (OuterVolumeSpecName: "home") pod "c7d8e4c2-4057-4b20-8874-20873ed0ff7f" (UID: "c7d8e4c2-4057-4b20-8874-20873ed0ff7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:12:00.965284 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.965232    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7d8e4c2-4057-4b20-8874-20873ed0ff7f" (UID: "c7d8e4c2-4057-4b20-8874-20873ed0ff7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:12:00.965284 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.965245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7d8e4c2-4057-4b20-8874-20873ed0ff7f" (UID: "c7d8e4c2-4057-4b20-8874-20873ed0ff7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:12:00.965284 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.965242    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-home\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:12:00.967220 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.967199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7d8e4c2-4057-4b20-8874-20873ed0ff7f" (UID: "c7d8e4c2-4057-4b20-8874-20873ed0ff7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:12:00.967324 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.967229    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-dshm" (OuterVolumeSpecName: "dshm") pod "c7d8e4c2-4057-4b20-8874-20873ed0ff7f" (UID: "c7d8e4c2-4057-4b20-8874-20873ed0ff7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:12:00.967324 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:00.967266    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-kube-api-access-m7kmb" (OuterVolumeSpecName: "kube-api-access-m7kmb") pod "c7d8e4c2-4057-4b20-8874-20873ed0ff7f" (UID: "c7d8e4c2-4057-4b20-8874-20873ed0ff7f"). InnerVolumeSpecName "kube-api-access-m7kmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:12:01.065820 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.065780    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-dshm\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:12:01.065820 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.065812    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tls-certs\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:12:01.065820 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.065823    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7kmb\" (UniqueName: \"kubernetes.io/projected/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-kube-api-access-m7kmb\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:12:01.066086 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.065834    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-tmp-dir\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:12:01.066086 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.065842    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d8e4c2-4057-4b20-8874-20873ed0ff7f-model-cache\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:12:01.536728 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.536691    2583 generic.go:356] "Generic (PLEG): container finished" podID="c7d8e4c2-4057-4b20-8874-20873ed0ff7f" containerID="b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869" exitCode=0
Jul 22 03:12:01.536938 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.536735    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"
Jul 22 03:12:01.536938 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.536778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs" event={"ID":"c7d8e4c2-4057-4b20-8874-20873ed0ff7f","Type":"ContainerDied","Data":"b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869"}
Jul 22 03:12:01.536938 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.536819    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs" event={"ID":"c7d8e4c2-4057-4b20-8874-20873ed0ff7f","Type":"ContainerDied","Data":"da7f38deea703b78bfb1158fa1499145060cbdcbb9af929e2c22982e7b4db37b"}
Jul 22 03:12:01.536938 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.536836    2583 scope.go:117] "RemoveContainer" containerID="b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869"
Jul 22 03:12:01.547822 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.547804    2583 scope.go:117] "RemoveContainer" containerID="b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869"
Jul 22 03:12:01.548111 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:12:01.548091    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869\": container with ID starting with b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869 not found: ID does not exist" containerID="b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869"
Jul 22 03:12:01.548166 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.548121    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869"} err="failed to get container status \"b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869\": rpc error: code = NotFound desc = could not find container \"b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869\": container with ID starting with b3eb89211c0a767533ebb7877f40f581559894c112211f8d55edf3d8f3372869 not found: ID does not exist"
Jul 22 03:12:01.560793 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.560770    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"]
Jul 22 03:12:01.566394 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:01.566368    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-e6d029/sub-test-model-e6d029-kserve-796cf85499-wgkgs"]
Jul 22 03:12:02.869428 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:02.869398    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d8e4c2-4057-4b20-8874-20873ed0ff7f" path="/var/lib/kubelet/pods/c7d8e4c2-4057-4b20-8874-20873ed0ff7f/volumes"
Jul 22 03:12:05.802131 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:05.802085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:12:05.802131 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:05.802135    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:12:05.814776 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:05.814744    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:12:06.571639 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:06.571611    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:12:33.180558 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:12:33.180526    2583 scope.go:117] "RemoveContainer" containerID="0aacd70116aece25be2bead5a31891b079170774e56751217547ac04ba3b1249"
Jul 22 03:13:07.834070 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:07.834020    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"]
Jul 22 03:13:09.682064 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.682018    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7912-140717351308976"]
Jul 22 03:13:09.682470 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.682414    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d8e4c2-4057-4b20-8874-20873ed0ff7f" containerName="main"
Jul 22 03:13:09.682470 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.682426    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d8e4c2-4057-4b20-8874-20873ed0ff7f" containerName="main"
Jul 22 03:13:09.682562 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.682497    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d8e4c2-4057-4b20-8874-20873ed0ff7f" containerName="main"
Jul 22 03:13:09.685563 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.685545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351308976"
Jul 22 03:13:09.688373 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.688352    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vnmk7\""
Jul 22 03:13:09.691924 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.691899    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351308976"]
Jul 22 03:13:09.793397 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.793366    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x95vd\" (UniqueName: \"kubernetes.io/projected/81f02f04-3f77-4a10-b66d-7a1405987aec-kube-api-access-x95vd\") pod \"test-curl-7912-140717351308976\" (UID: \"81f02f04-3f77-4a10-b66d-7a1405987aec\") " pod="opendatahub/test-curl-7912-140717351308976"
Jul 22 03:13:09.894414 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.894376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x95vd\" (UniqueName: \"kubernetes.io/projected/81f02f04-3f77-4a10-b66d-7a1405987aec-kube-api-access-x95vd\") pod \"test-curl-7912-140717351308976\" (UID: \"81f02f04-3f77-4a10-b66d-7a1405987aec\") " pod="opendatahub/test-curl-7912-140717351308976"
Jul 22 03:13:09.905046 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.904993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x95vd\" (UniqueName: \"kubernetes.io/projected/81f02f04-3f77-4a10-b66d-7a1405987aec-kube-api-access-x95vd\") pod \"test-curl-7912-140717351308976\" (UID: \"81f02f04-3f77-4a10-b66d-7a1405987aec\") " pod="opendatahub/test-curl-7912-140717351308976"
Jul 22 03:13:09.996038 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:09.995996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351308976"
Jul 22 03:13:10.130093 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:10.130068    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351308976"]
Jul 22 03:13:10.132154 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:13:10.132118    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81f02f04_3f77_4a10_b66d_7a1405987aec.slice/crio-06a4933bcff512775666f4702c33325031ebbea08288b482da9e4a288b163c3a WatchSource:0}: Error finding container 06a4933bcff512775666f4702c33325031ebbea08288b482da9e4a288b163c3a: Status 404 returned error can't find the container with id 06a4933bcff512775666f4702c33325031ebbea08288b482da9e4a288b163c3a
Jul 22 03:13:10.133337 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:10.133316    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 03:13:10.868989 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:10.868959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351308976" event={"ID":"81f02f04-3f77-4a10-b66d-7a1405987aec","Type":"ContainerStarted","Data":"06a4933bcff512775666f4702c33325031ebbea08288b482da9e4a288b163c3a"}
Jul 22 03:13:11.874741 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:11.874702    2583 generic.go:356] "Generic (PLEG): container finished" podID="81f02f04-3f77-4a10-b66d-7a1405987aec" containerID="ca643fab00f824a5bfa8c0ac2578a2c79a4282a558fd299c96dc60806ec78958" exitCode=0
Jul 22 03:13:11.875151 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:11.874775    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351308976" event={"ID":"81f02f04-3f77-4a10-b66d-7a1405987aec","Type":"ContainerDied","Data":"ca643fab00f824a5bfa8c0ac2578a2c79a4282a558fd299c96dc60806ec78958"}
Jul 22 03:13:13.010328 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.010303    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351308976"
Jul 22 03:13:13.048299 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.048272    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7912-140717351308976_81f02f04-3f77-4a10-b66d-7a1405987aec/test-curl-7912-140717351308976/0.log"
Jul 22 03:13:13.072911 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.072876    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7912-140717351308976"]
Jul 22 03:13:13.076512 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.076489    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7912-140717351308976"]
Jul 22 03:13:13.123167 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.123136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x95vd\" (UniqueName: \"kubernetes.io/projected/81f02f04-3f77-4a10-b66d-7a1405987aec-kube-api-access-x95vd\") pod \"81f02f04-3f77-4a10-b66d-7a1405987aec\" (UID: \"81f02f04-3f77-4a10-b66d-7a1405987aec\") "
Jul 22 03:13:13.125500 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.125470    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f02f04-3f77-4a10-b66d-7a1405987aec-kube-api-access-x95vd" (OuterVolumeSpecName: "kube-api-access-x95vd") pod "81f02f04-3f77-4a10-b66d-7a1405987aec" (UID: "81f02f04-3f77-4a10-b66d-7a1405987aec"). InnerVolumeSpecName "kube-api-access-x95vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:13:13.201146 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.201108    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7912-140717351306800"]
Jul 22 03:13:13.201458 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.201444    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f02f04-3f77-4a10-b66d-7a1405987aec" containerName="test-curl-7912-140717351308976"
Jul 22 03:13:13.201501 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.201460    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f02f04-3f77-4a10-b66d-7a1405987aec" containerName="test-curl-7912-140717351308976"
Jul 22 03:13:13.201545 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.201536    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f02f04-3f77-4a10-b66d-7a1405987aec" containerName="test-curl-7912-140717351308976"
Jul 22 03:13:13.204417 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.204397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351306800"
Jul 22 03:13:13.211625 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.211594    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351306800"]
Jul 22 03:13:13.224152 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.224122    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x95vd\" (UniqueName: \"kubernetes.io/projected/81f02f04-3f77-4a10-b66d-7a1405987aec-kube-api-access-x95vd\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:13.325258 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.325221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp9sf\" (UniqueName: \"kubernetes.io/projected/371351b1-9be2-411a-8f71-1f12aaf56b73-kube-api-access-dp9sf\") pod \"test-curl-7912-140717351306800\" (UID: \"371351b1-9be2-411a-8f71-1f12aaf56b73\") " pod="opendatahub/test-curl-7912-140717351306800"
Jul 22 03:13:13.426574 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.426460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp9sf\" (UniqueName: \"kubernetes.io/projected/371351b1-9be2-411a-8f71-1f12aaf56b73-kube-api-access-dp9sf\") pod \"test-curl-7912-140717351306800\" (UID: \"371351b1-9be2-411a-8f71-1f12aaf56b73\") " pod="opendatahub/test-curl-7912-140717351306800"
Jul 22 03:13:13.435395 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.435366    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp9sf\" (UniqueName: \"kubernetes.io/projected/371351b1-9be2-411a-8f71-1f12aaf56b73-kube-api-access-dp9sf\") pod \"test-curl-7912-140717351306800\" (UID: \"371351b1-9be2-411a-8f71-1f12aaf56b73\") " pod="opendatahub/test-curl-7912-140717351306800"
Jul 22 03:13:13.515104 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.515075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351306800"
Jul 22 03:13:13.646416 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.646390    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351306800"]
Jul 22 03:13:13.648919 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:13:13.648886    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod371351b1_9be2_411a_8f71_1f12aaf56b73.slice/crio-a126b22f8d34f707c778e71112be0aa7b56d39dbe355fb98f4f65e0819f01de6 WatchSource:0}: Error finding container a126b22f8d34f707c778e71112be0aa7b56d39dbe355fb98f4f65e0819f01de6: Status 404 returned error can't find the container with id a126b22f8d34f707c778e71112be0aa7b56d39dbe355fb98f4f65e0819f01de6
Jul 22 03:13:13.885177 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.885142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351306800" event={"ID":"371351b1-9be2-411a-8f71-1f12aaf56b73","Type":"ContainerStarted","Data":"a126b22f8d34f707c778e71112be0aa7b56d39dbe355fb98f4f65e0819f01de6"}
Jul 22 03:13:13.886347 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.886327    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06a4933bcff512775666f4702c33325031ebbea08288b482da9e4a288b163c3a"
Jul 22 03:13:13.886460 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:13.886368    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351308976"
Jul 22 03:13:13.897454 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:13:13.897425    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7912-140717351308976\" is forbidden: User \"system:node:ip-10-0-133-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-30.ec2.internal' and this object" podUID="81f02f04-3f77-4a10-b66d-7a1405987aec" pod="opendatahub/test-curl-7912-140717351308976"
Jul 22 03:13:14.875601 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:14.875560    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f02f04-3f77-4a10-b66d-7a1405987aec" path="/var/lib/kubelet/pods/81f02f04-3f77-4a10-b66d-7a1405987aec/volumes"
Jul 22 03:13:14.891717 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:14.891683    2583 generic.go:356] "Generic (PLEG): container finished" podID="371351b1-9be2-411a-8f71-1f12aaf56b73" containerID="1e3930e18fe3c07661a82941f832c5ccfe1caca61a1a6856a290887cbadf05d5" exitCode=0
Jul 22 03:13:14.891873 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:14.891761    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351306800" event={"ID":"371351b1-9be2-411a-8f71-1f12aaf56b73","Type":"ContainerDied","Data":"1e3930e18fe3c07661a82941f832c5ccfe1caca61a1a6856a290887cbadf05d5"}
Jul 22 03:13:16.028976 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.028952    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351306800"
Jul 22 03:13:16.050653 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.050627    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7912-140717351306800_371351b1-9be2-411a-8f71-1f12aaf56b73/test-curl-7912-140717351306800/0.log"
Jul 22 03:13:16.076603 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.076571    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7912-140717351306800"]
Jul 22 03:13:16.080499 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.080476    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7912-140717351306800"]
Jul 22 03:13:16.151014 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.150939    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp9sf\" (UniqueName: \"kubernetes.io/projected/371351b1-9be2-411a-8f71-1f12aaf56b73-kube-api-access-dp9sf\") pod \"371351b1-9be2-411a-8f71-1f12aaf56b73\" (UID: \"371351b1-9be2-411a-8f71-1f12aaf56b73\") "
Jul 22 03:13:16.153192 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.153164    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371351b1-9be2-411a-8f71-1f12aaf56b73-kube-api-access-dp9sf" (OuterVolumeSpecName: "kube-api-access-dp9sf") pod "371351b1-9be2-411a-8f71-1f12aaf56b73" (UID: "371351b1-9be2-411a-8f71-1f12aaf56b73"). InnerVolumeSpecName "kube-api-access-dp9sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:13:16.252511 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.252473    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp9sf\" (UniqueName: \"kubernetes.io/projected/371351b1-9be2-411a-8f71-1f12aaf56b73-kube-api-access-dp9sf\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:16.868363 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.868328    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371351b1-9be2-411a-8f71-1f12aaf56b73" path="/var/lib/kubelet/pods/371351b1-9be2-411a-8f71-1f12aaf56b73/volumes"
Jul 22 03:13:16.901714 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.901691    2583 scope.go:117] "RemoveContainer" containerID="1e3930e18fe3c07661a82941f832c5ccfe1caca61a1a6856a290887cbadf05d5"
Jul 22 03:13:16.901904 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:16.901689    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351306800"
Jul 22 03:13:18.957974 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:18.957936    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7912-140717351253024"]
Jul 22 03:13:18.958374 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:18.958330    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371351b1-9be2-411a-8f71-1f12aaf56b73" containerName="test-curl-7912-140717351306800"
Jul 22 03:13:18.958374 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:18.958344    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="371351b1-9be2-411a-8f71-1f12aaf56b73" containerName="test-curl-7912-140717351306800"
Jul 22 03:13:18.958444 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:18.958402    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="371351b1-9be2-411a-8f71-1f12aaf56b73" containerName="test-curl-7912-140717351306800"
Jul 22 03:13:18.962834 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:18.962817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351253024"
Jul 22 03:13:18.965275 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:18.965197    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vnmk7\""
Jul 22 03:13:18.967342 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:18.967317    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351253024"]
Jul 22 03:13:19.076842 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:19.076810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpqh\" (UniqueName: \"kubernetes.io/projected/3abfd786-85b1-4d1d-affa-d10a1c0b76ad-kube-api-access-7rpqh\") pod \"test-curl-7912-140717351253024\" (UID: \"3abfd786-85b1-4d1d-affa-d10a1c0b76ad\") " pod="opendatahub/test-curl-7912-140717351253024"
Jul 22 03:13:19.178380 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:19.178341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpqh\" (UniqueName: \"kubernetes.io/projected/3abfd786-85b1-4d1d-affa-d10a1c0b76ad-kube-api-access-7rpqh\") pod \"test-curl-7912-140717351253024\" (UID: \"3abfd786-85b1-4d1d-affa-d10a1c0b76ad\") " pod="opendatahub/test-curl-7912-140717351253024"
Jul 22 03:13:19.186424 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:19.186398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpqh\" (UniqueName: \"kubernetes.io/projected/3abfd786-85b1-4d1d-affa-d10a1c0b76ad-kube-api-access-7rpqh\") pod \"test-curl-7912-140717351253024\" (UID: \"3abfd786-85b1-4d1d-affa-d10a1c0b76ad\") " pod="opendatahub/test-curl-7912-140717351253024"
Jul 22 03:13:19.272923 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:19.272884    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351253024"
Jul 22 03:13:19.408821 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:19.408780    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351253024"]
Jul 22 03:13:19.411245 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:13:19.411221    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3abfd786_85b1_4d1d_affa_d10a1c0b76ad.slice/crio-a3941c8182f1adb6d4d187694b6419af32b0707c4527d6db8d54297c16229f0a WatchSource:0}: Error finding container a3941c8182f1adb6d4d187694b6419af32b0707c4527d6db8d54297c16229f0a: Status 404 returned error can't find the container with id a3941c8182f1adb6d4d187694b6419af32b0707c4527d6db8d54297c16229f0a
Jul 22 03:13:19.921163 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:19.921123    2583 generic.go:356] "Generic (PLEG): container finished" podID="3abfd786-85b1-4d1d-affa-d10a1c0b76ad" containerID="35fbb709ff56abcec0a01e103298714f1cf9b359d7b5edcd8b27c8e0f3e38042" exitCode=0
Jul 22 03:13:19.921346 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:19.921174    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351253024" event={"ID":"3abfd786-85b1-4d1d-affa-d10a1c0b76ad","Type":"ContainerDied","Data":"35fbb709ff56abcec0a01e103298714f1cf9b359d7b5edcd8b27c8e0f3e38042"}
Jul 22 03:13:19.921346 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:19.921204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351253024" event={"ID":"3abfd786-85b1-4d1d-affa-d10a1c0b76ad","Type":"ContainerStarted","Data":"a3941c8182f1adb6d4d187694b6419af32b0707c4527d6db8d54297c16229f0a"}
Jul 22 03:13:21.056234 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.056210    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351253024"
Jul 22 03:13:21.077870 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.077838    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7912-140717351253024_3abfd786-85b1-4d1d-affa-d10a1c0b76ad/test-curl-7912-140717351253024/0.log"
Jul 22 03:13:21.102574 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.102545    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7912-140717351253024"]
Jul 22 03:13:21.107323 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.107299    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7912-140717351253024"]
Jul 22 03:13:21.198406 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.198326    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rpqh\" (UniqueName: \"kubernetes.io/projected/3abfd786-85b1-4d1d-affa-d10a1c0b76ad-kube-api-access-7rpqh\") pod \"3abfd786-85b1-4d1d-affa-d10a1c0b76ad\" (UID: \"3abfd786-85b1-4d1d-affa-d10a1c0b76ad\") "
Jul 22 03:13:21.200512 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.200484    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abfd786-85b1-4d1d-affa-d10a1c0b76ad-kube-api-access-7rpqh" (OuterVolumeSpecName: "kube-api-access-7rpqh") pod "3abfd786-85b1-4d1d-affa-d10a1c0b76ad" (UID: "3abfd786-85b1-4d1d-affa-d10a1c0b76ad"). InnerVolumeSpecName "kube-api-access-7rpqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:13:21.231992 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.231955    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7912-140717351254032"]
Jul 22 03:13:21.232349 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.232336    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3abfd786-85b1-4d1d-affa-d10a1c0b76ad" containerName="test-curl-7912-140717351253024"
Jul 22 03:13:21.232399 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.232352    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3abfd786-85b1-4d1d-affa-d10a1c0b76ad" containerName="test-curl-7912-140717351253024"
Jul 22 03:13:21.232435 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.232428    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3abfd786-85b1-4d1d-affa-d10a1c0b76ad" containerName="test-curl-7912-140717351253024"
Jul 22 03:13:21.235426 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.235410    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351254032"
Jul 22 03:13:21.240393 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.240359    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351254032"]
Jul 22 03:13:21.299572 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.299539    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rpqh\" (UniqueName: \"kubernetes.io/projected/3abfd786-85b1-4d1d-affa-d10a1c0b76ad-kube-api-access-7rpqh\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:21.400466 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.400431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhtg\" (UniqueName: \"kubernetes.io/projected/7119e3aa-3e3d-408f-b574-bf45d6ecb7d4-kube-api-access-nxhtg\") pod \"test-curl-7912-140717351254032\" (UID: \"7119e3aa-3e3d-408f-b574-bf45d6ecb7d4\") " pod="opendatahub/test-curl-7912-140717351254032"
Jul 22 03:13:21.501189 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.501157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhtg\" (UniqueName: \"kubernetes.io/projected/7119e3aa-3e3d-408f-b574-bf45d6ecb7d4-kube-api-access-nxhtg\") pod \"test-curl-7912-140717351254032\" (UID: \"7119e3aa-3e3d-408f-b574-bf45d6ecb7d4\") " pod="opendatahub/test-curl-7912-140717351254032"
Jul 22 03:13:21.509271 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.509242    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhtg\" (UniqueName: \"kubernetes.io/projected/7119e3aa-3e3d-408f-b574-bf45d6ecb7d4-kube-api-access-nxhtg\") pod \"test-curl-7912-140717351254032\" (UID: \"7119e3aa-3e3d-408f-b574-bf45d6ecb7d4\") " pod="opendatahub/test-curl-7912-140717351254032"
Jul 22 03:13:21.546358 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.546319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351254032"
Jul 22 03:13:21.678509 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.678483    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7912-140717351254032"]
Jul 22 03:13:21.680880 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:13:21.680848    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7119e3aa_3e3d_408f_b574_bf45d6ecb7d4.slice/crio-5bae2e3bd9dd15fd595bef9c195504f6e2c5f512dd8ad0f6df7d8f9a1d03effa WatchSource:0}: Error finding container 5bae2e3bd9dd15fd595bef9c195504f6e2c5f512dd8ad0f6df7d8f9a1d03effa: Status 404 returned error can't find the container with id 5bae2e3bd9dd15fd595bef9c195504f6e2c5f512dd8ad0f6df7d8f9a1d03effa
Jul 22 03:13:21.932129 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.932045    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351254032" event={"ID":"7119e3aa-3e3d-408f-b574-bf45d6ecb7d4","Type":"ContainerStarted","Data":"5bae2e3bd9dd15fd595bef9c195504f6e2c5f512dd8ad0f6df7d8f9a1d03effa"}
Jul 22 03:13:21.933190 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.933170    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351253024"
Jul 22 03:13:21.933190 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:21.933183    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3941c8182f1adb6d4d187694b6419af32b0707c4527d6db8d54297c16229f0a"
Jul 22 03:13:21.942904 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:13:21.942874    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7912-140717351253024\" is forbidden: User \"system:node:ip-10-0-133-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-30.ec2.internal' and this object" podUID="3abfd786-85b1-4d1d-affa-d10a1c0b76ad" pod="opendatahub/test-curl-7912-140717351253024"
Jul 22 03:13:22.854827 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:22.854782    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r" podUID="746176e7-6341-4b6c-8edb-8b072e0445a3" containerName="main" containerID="cri-o://a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a" gracePeriod=45
Jul 22 03:13:22.869617 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:22.869590    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3abfd786-85b1-4d1d-affa-d10a1c0b76ad" path="/var/lib/kubelet/pods/3abfd786-85b1-4d1d-affa-d10a1c0b76ad/volumes"
Jul 22 03:13:22.939233 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:22.939196    2583 generic.go:356] "Generic (PLEG): container finished" podID="7119e3aa-3e3d-408f-b574-bf45d6ecb7d4" containerID="a3386a19c7b9b055914097305d9341c2ec8c09651bb6ae8e2276ceb3194943f3" exitCode=0
Jul 22 03:13:22.939370 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:22.939282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7912-140717351254032" event={"ID":"7119e3aa-3e3d-408f-b574-bf45d6ecb7d4","Type":"ContainerDied","Data":"a3386a19c7b9b055914097305d9341c2ec8c09651bb6ae8e2276ceb3194943f3"}
Jul 22 03:13:23.106629 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.106567    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:13:23.217461 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.217428    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4dcr\" (UniqueName: \"kubernetes.io/projected/746176e7-6341-4b6c-8edb-8b072e0445a3-kube-api-access-w4dcr\") pod \"746176e7-6341-4b6c-8edb-8b072e0445a3\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") "
Jul 22 03:13:23.217642 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.217482    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746176e7-6341-4b6c-8edb-8b072e0445a3-tls-certs\") pod \"746176e7-6341-4b6c-8edb-8b072e0445a3\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") "
Jul 22 03:13:23.217642 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.217566    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-dshm\") pod \"746176e7-6341-4b6c-8edb-8b072e0445a3\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") "
Jul 22 03:13:23.217642 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.217587    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-tmp-dir\") pod \"746176e7-6341-4b6c-8edb-8b072e0445a3\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") "
Jul 22 03:13:23.217642 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.217609    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-home\") pod \"746176e7-6341-4b6c-8edb-8b072e0445a3\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") "
Jul 22 03:13:23.217642 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.217636    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-model-cache\") pod \"746176e7-6341-4b6c-8edb-8b072e0445a3\" (UID: \"746176e7-6341-4b6c-8edb-8b072e0445a3\") "
Jul 22 03:13:23.217902 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.217868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "746176e7-6341-4b6c-8edb-8b072e0445a3" (UID: "746176e7-6341-4b6c-8edb-8b072e0445a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:13:23.217902 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.217887    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-home" (OuterVolumeSpecName: "home") pod "746176e7-6341-4b6c-8edb-8b072e0445a3" (UID: "746176e7-6341-4b6c-8edb-8b072e0445a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:13:23.218049 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.218009    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "746176e7-6341-4b6c-8edb-8b072e0445a3" (UID: "746176e7-6341-4b6c-8edb-8b072e0445a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:13:23.219924 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.219892    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746176e7-6341-4b6c-8edb-8b072e0445a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "746176e7-6341-4b6c-8edb-8b072e0445a3" (UID: "746176e7-6341-4b6c-8edb-8b072e0445a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:13:23.220089 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.219926    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-dshm" (OuterVolumeSpecName: "dshm") pod "746176e7-6341-4b6c-8edb-8b072e0445a3" (UID: "746176e7-6341-4b6c-8edb-8b072e0445a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 03:13:23.220089 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.219998    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746176e7-6341-4b6c-8edb-8b072e0445a3-kube-api-access-w4dcr" (OuterVolumeSpecName: "kube-api-access-w4dcr") pod "746176e7-6341-4b6c-8edb-8b072e0445a3" (UID: "746176e7-6341-4b6c-8edb-8b072e0445a3"). InnerVolumeSpecName "kube-api-access-w4dcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:13:23.318424 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.318394    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-dshm\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:23.318424 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.318420    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-tmp-dir\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:23.318424 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.318429    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-home\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:23.318424 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.318439    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/746176e7-6341-4b6c-8edb-8b072e0445a3-model-cache\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:23.318705 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.318448    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dcr\" (UniqueName: \"kubernetes.io/projected/746176e7-6341-4b6c-8edb-8b072e0445a3-kube-api-access-w4dcr\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:23.318705 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.318457    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/746176e7-6341-4b6c-8edb-8b072e0445a3-tls-certs\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:23.945167 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.945127    2583 generic.go:356] "Generic (PLEG): container finished" podID="746176e7-6341-4b6c-8edb-8b072e0445a3" containerID="a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a" exitCode=0
Jul 22 03:13:23.945651 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.945176    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"
Jul 22 03:13:23.945651 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.945212    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r" event={"ID":"746176e7-6341-4b6c-8edb-8b072e0445a3","Type":"ContainerDied","Data":"a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a"}
Jul 22 03:13:23.945651 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.945260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r" event={"ID":"746176e7-6341-4b6c-8edb-8b072e0445a3","Type":"ContainerDied","Data":"5877ad7b6dce477cf082798462e3ad0612fdf137adb036d2443ead0d5e1ed68e"}
Jul 22 03:13:23.945651 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.945289    2583 scope.go:117] "RemoveContainer" containerID="a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a"
Jul 22 03:13:23.960970 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.960950    2583 scope.go:117] "RemoveContainer" containerID="a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a"
Jul 22 03:13:23.961377 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:13:23.961353    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a\": container with ID starting with a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a not found: ID does not exist" containerID="a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a"
Jul 22 03:13:23.961503 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.961386    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a"} err="failed to get container status \"a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a\": rpc error: code = NotFound desc = could not find container \"a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a\": container with ID starting with a80b1e1ebfa75279ff66a526af63fd795733ade8508b8fd280d38dfbe0e1483a not found: ID does not exist"
Jul 22 03:13:23.971004 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.970970    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"]
Jul 22 03:13:23.975252 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:23.975222    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-643135/rate-test-model-643135-kserve-77b688d779-ghk8r"]
Jul 22 03:13:24.084128 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.084067    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351254032"
Jul 22 03:13:24.104989 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.104963    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7912-140717351254032_7119e3aa-3e3d-408f-b574-bf45d6ecb7d4/test-curl-7912-140717351254032/0.log"
Jul 22 03:13:24.131918 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.131890    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7912-140717351254032"]
Jul 22 03:13:24.136190 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.136163    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7912-140717351254032"]
Jul 22 03:13:24.227083 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.226974    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxhtg\" (UniqueName: \"kubernetes.io/projected/7119e3aa-3e3d-408f-b574-bf45d6ecb7d4-kube-api-access-nxhtg\") pod \"7119e3aa-3e3d-408f-b574-bf45d6ecb7d4\" (UID: \"7119e3aa-3e3d-408f-b574-bf45d6ecb7d4\") "
Jul 22 03:13:24.229249 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.229222    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7119e3aa-3e3d-408f-b574-bf45d6ecb7d4-kube-api-access-nxhtg" (OuterVolumeSpecName: "kube-api-access-nxhtg") pod "7119e3aa-3e3d-408f-b574-bf45d6ecb7d4" (UID: "7119e3aa-3e3d-408f-b574-bf45d6ecb7d4"). InnerVolumeSpecName "kube-api-access-nxhtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:13:24.327996 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.327952    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxhtg\" (UniqueName: \"kubernetes.io/projected/7119e3aa-3e3d-408f-b574-bf45d6ecb7d4-kube-api-access-nxhtg\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:24.868388 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.868357    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7119e3aa-3e3d-408f-b574-bf45d6ecb7d4" path="/var/lib/kubelet/pods/7119e3aa-3e3d-408f-b574-bf45d6ecb7d4/volumes"
Jul 22 03:13:24.868737 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.868725    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746176e7-6341-4b6c-8edb-8b072e0445a3" path="/var/lib/kubelet/pods/746176e7-6341-4b6c-8edb-8b072e0445a3/volumes"
Jul 22 03:13:24.950821 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.950789    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7912-140717351254032"
Jul 22 03:13:24.951275 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:24.950823    2583 scope.go:117] "RemoveContainer" containerID="a3386a19c7b9b055914097305d9341c2ec8c09651bb6ae8e2276ceb3194943f3"
Jul 22 03:13:27.272059 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.272013    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"]
Jul 22 03:13:27.272423 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.272388    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7119e3aa-3e3d-408f-b574-bf45d6ecb7d4" containerName="test-curl-7912-140717351254032"
Jul 22 03:13:27.272423 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.272400    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7119e3aa-3e3d-408f-b574-bf45d6ecb7d4" containerName="test-curl-7912-140717351254032"
Jul 22 03:13:27.272423 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.272415    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746176e7-6341-4b6c-8edb-8b072e0445a3" containerName="main"
Jul 22 03:13:27.272423 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.272421    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="746176e7-6341-4b6c-8edb-8b072e0445a3" containerName="main"
Jul 22 03:13:27.272578 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.272484    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7119e3aa-3e3d-408f-b574-bf45d6ecb7d4" containerName="test-curl-7912-140717351254032"
Jul 22 03:13:27.272578 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.272492    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="746176e7-6341-4b6c-8edb-8b072e0445a3" containerName="main"
Jul 22 03:13:27.276465 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.276445    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"
Jul 22 03:13:27.279021 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.279001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\""
Jul 22 03:13:27.281943 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.281923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"]
Jul 22 03:13:27.454937 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.454894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0c711-464a-44dc-95b0-aa32ba7f33a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-643135-n8v9z\" (UID: \"77d0c711-464a-44dc-95b0-aa32ba7f33a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"
Jul 22 03:13:27.556505 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.556415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0c711-464a-44dc-95b0-aa32ba7f33a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-643135-n8v9z\" (UID: \"77d0c711-464a-44dc-95b0-aa32ba7f33a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"
Jul 22 03:13:27.557001 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.556983    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0c711-464a-44dc-95b0-aa32ba7f33a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-643135-n8v9z\" (UID: \"77d0c711-464a-44dc-95b0-aa32ba7f33a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"
Jul 22 03:13:27.587836 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.587803    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"
Jul 22 03:13:27.723637 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.723614    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"]
Jul 22 03:13:27.726437 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:13:27.726406    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d0c711_464a_44dc_95b0_aa32ba7f33a4.slice/crio-d77925db1eae2587a6ec96920b73c59d0a7ae68f30f0364c89ad458789117da6 WatchSource:0}: Error finding container d77925db1eae2587a6ec96920b73c59d0a7ae68f30f0364c89ad458789117da6: Status 404 returned error can't find the container with id d77925db1eae2587a6ec96920b73c59d0a7ae68f30f0364c89ad458789117da6
Jul 22 03:13:27.968236 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:27.968199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z" event={"ID":"77d0c711-464a-44dc-95b0-aa32ba7f33a4","Type":"ContainerStarted","Data":"d77925db1eae2587a6ec96920b73c59d0a7ae68f30f0364c89ad458789117da6"}
Jul 22 03:13:28.974626 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:28.974591    2583 generic.go:356] "Generic (PLEG): container finished" podID="77d0c711-464a-44dc-95b0-aa32ba7f33a4" containerID="b513f0d63ebb38c3ea38e988f3339ed4f991a12779aeaf53541bbe30a088579f" exitCode=0
Jul 22 03:13:28.975134 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:28.974632    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z" event={"ID":"77d0c711-464a-44dc-95b0-aa32ba7f33a4","Type":"ContainerDied","Data":"b513f0d63ebb38c3ea38e988f3339ed4f991a12779aeaf53541bbe30a088579f"}
Jul 22 03:13:30.112823 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.112801    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"
Jul 22 03:13:30.178907 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.178869    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0c711-464a-44dc-95b0-aa32ba7f33a4-maas-api-service-ca\") pod \"77d0c711-464a-44dc-95b0-aa32ba7f33a4\" (UID: \"77d0c711-464a-44dc-95b0-aa32ba7f33a4\") "
Jul 22 03:13:30.179264 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.179242    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d0c711-464a-44dc-95b0-aa32ba7f33a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77d0c711-464a-44dc-95b0-aa32ba7f33a4" (UID: "77d0c711-464a-44dc-95b0-aa32ba7f33a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:13:30.279870 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.279789    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d0c711-464a-44dc-95b0-aa32ba7f33a4-maas-api-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:30.946045 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.946001    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 03:13:30.949451 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.949421    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xxrcb_b75189cc-3291-4d83-85d9-c5590098fea5/ovn-acl-logging/0.log"
Jul 22 03:13:30.986558 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.986525    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"
Jul 22 03:13:30.986710 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.986558    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z" event={"ID":"77d0c711-464a-44dc-95b0-aa32ba7f33a4","Type":"ContainerDied","Data":"d77925db1eae2587a6ec96920b73c59d0a7ae68f30f0364c89ad458789117da6"}
Jul 22 03:13:30.986710 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:30.986615    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d77925db1eae2587a6ec96920b73c59d0a7ae68f30f0364c89ad458789117da6"
Jul 22 03:13:33.203206 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:33.203166    2583 scope.go:117] "RemoveContainer" containerID="725750f4d6b34acdb96c9b00e3562893841b2131dfa6cca29d59c660adc0d00d"
Jul 22 03:13:33.211499 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:33.211479    2583 scope.go:117] "RemoveContainer" containerID="184e2d8008f1bd4a1099791b1787e84e2fe7538a61796a5bf2d2996eec802e1f"
Jul 22 03:13:37.984078 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:37.984044    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"]
Jul 22 03:13:37.984489 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:37.984232    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4" podUID="d9074df5-289c-4035-a283-78c13fdb756c" containerName="maas-api" containerID="cri-o://ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e" gracePeriod=30
Jul 22 03:13:38.240713 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.240641    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:13:38.342679 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.342645    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j222k\" (UniqueName: \"kubernetes.io/projected/d9074df5-289c-4035-a283-78c13fdb756c-kube-api-access-j222k\") pod \"d9074df5-289c-4035-a283-78c13fdb756c\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") "
Jul 22 03:13:38.342871 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.342688    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9074df5-289c-4035-a283-78c13fdb756c-openshift-service-ca\") pod \"d9074df5-289c-4035-a283-78c13fdb756c\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") "
Jul 22 03:13:38.342871 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.342728    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls\") pod \"d9074df5-289c-4035-a283-78c13fdb756c\" (UID: \"d9074df5-289c-4035-a283-78c13fdb756c\") "
Jul 22 03:13:38.343049 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.343008    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9074df5-289c-4035-a283-78c13fdb756c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9074df5-289c-4035-a283-78c13fdb756c" (UID: "d9074df5-289c-4035-a283-78c13fdb756c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:13:38.344987 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.344963    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9074df5-289c-4035-a283-78c13fdb756c" (UID: "d9074df5-289c-4035-a283-78c13fdb756c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:13:38.345060 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.345012    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9074df5-289c-4035-a283-78c13fdb756c-kube-api-access-j222k" (OuterVolumeSpecName: "kube-api-access-j222k") pod "d9074df5-289c-4035-a283-78c13fdb756c" (UID: "d9074df5-289c-4035-a283-78c13fdb756c"). InnerVolumeSpecName "kube-api-access-j222k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:13:38.444179 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.444143    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j222k\" (UniqueName: \"kubernetes.io/projected/d9074df5-289c-4035-a283-78c13fdb756c-kube-api-access-j222k\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:38.444179 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.444174    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9074df5-289c-4035-a283-78c13fdb756c-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:38.444179 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:38.444184    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9074df5-289c-4035-a283-78c13fdb756c-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:39.025380 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.025342    2583 generic.go:356] "Generic (PLEG): container finished" podID="d9074df5-289c-4035-a283-78c13fdb756c" containerID="ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e" exitCode=0
Jul 22 03:13:39.025910 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.025389    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"
Jul 22 03:13:39.025910 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.025383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4" event={"ID":"d9074df5-289c-4035-a283-78c13fdb756c","Type":"ContainerDied","Data":"ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e"}
Jul 22 03:13:39.025910 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.025530    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4" event={"ID":"d9074df5-289c-4035-a283-78c13fdb756c","Type":"ContainerDied","Data":"f92823d5b7f8adfa98620011f13e3969bcc1ed3b78cb0808a42d3d8f7716aa1c"}
Jul 22 03:13:39.025910 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.025563    2583 scope.go:117] "RemoveContainer" containerID="ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e"
Jul 22 03:13:39.034135 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.034114    2583 scope.go:117] "RemoveContainer" containerID="ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e"
Jul 22 03:13:39.034417 ip-10-0-133-30 kubenswrapper[2583]: E0722 03:13:39.034390    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e\": container with ID starting with ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e not found: ID does not exist" containerID="ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e"
Jul 22 03:13:39.034529 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.034417    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e"} err="failed to get container status \"ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e\": rpc error: code = NotFound desc = could not find container \"ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e\": container with ID starting with ec0c0f0b8f628cdfc4cf01c996139459159f847a4bc30bb63a3fcd7b2b1eb47e not found: ID does not exist"
Jul 22 03:13:39.043234 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.043210    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"]
Jul 22 03:13:39.048266 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:39.048240    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-643135-6f87cdf6f9-b8dv4"]
Jul 22 03:13:40.877455 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:40.877412    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9074df5-289c-4035-a283-78c13fdb756c" path="/var/lib/kubelet/pods/d9074df5-289c-4035-a283-78c13fdb756c/volumes"
Jul 22 03:13:42.366043 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:42.366000    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"]
Jul 22 03:13:42.369024 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:42.369000    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-643135-n8v9z"]
Jul 22 03:13:42.869150 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:42.869113    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77d0c711-464a-44dc-95b0-aa32ba7f33a4" path="/var/lib/kubelet/pods/77d0c711-464a-44dc-95b0-aa32ba7f33a4/volumes"
Jul 22 03:13:49.337577 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.337544    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"]
Jul 22 03:13:49.337985 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.337892    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9074df5-289c-4035-a283-78c13fdb756c" containerName="maas-api"
Jul 22 03:13:49.337985 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.337902    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9074df5-289c-4035-a283-78c13fdb756c" containerName="maas-api"
Jul 22 03:13:49.337985 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.337911    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77d0c711-464a-44dc-95b0-aa32ba7f33a4" containerName="revoke-keys"
Jul 22 03:13:49.337985 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.337916    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="77d0c711-464a-44dc-95b0-aa32ba7f33a4" containerName="revoke-keys"
Jul 22 03:13:49.338136 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.337993    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9074df5-289c-4035-a283-78c13fdb756c" containerName="maas-api"
Jul 22 03:13:49.338136 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.338001    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="77d0c711-464a-44dc-95b0-aa32ba7f33a4" containerName="revoke-keys"
Jul 22 03:13:49.343215 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.343196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"
Jul 22 03:13:49.346071 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.346053    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tm89v\""
Jul 22 03:13:49.349725 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.349700    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"]
Jul 22 03:13:49.446543 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.446506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f95d49-a2f8-4333-b17b-83a4ef01ee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2\" (UID: \"e0f95d49-a2f8-4333-b17b-83a4ef01ee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"
Jul 22 03:13:49.547017 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.546980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f95d49-a2f8-4333-b17b-83a4ef01ee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2\" (UID: \"e0f95d49-a2f8-4333-b17b-83a4ef01ee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"
Jul 22 03:13:49.547630 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.547611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f95d49-a2f8-4333-b17b-83a4ef01ee05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2\" (UID: \"e0f95d49-a2f8-4333-b17b-83a4ef01ee05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"
Jul 22 03:13:49.654882 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.654793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"
Jul 22 03:13:49.787671 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:49.787650    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"]
Jul 22 03:13:49.789861 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:13:49.789835    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f95d49_a2f8_4333_b17b_83a4ef01ee05.slice/crio-f93825ddffe3c20eff74f8ecaf049de3da3b350fa6f5718686d792b69ca46801 WatchSource:0}: Error finding container f93825ddffe3c20eff74f8ecaf049de3da3b350fa6f5718686d792b69ca46801: Status 404 returned error can't find the container with id f93825ddffe3c20eff74f8ecaf049de3da3b350fa6f5718686d792b69ca46801
Jul 22 03:13:50.084270 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:50.084231    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2" event={"ID":"e0f95d49-a2f8-4333-b17b-83a4ef01ee05","Type":"ContainerStarted","Data":"cd1fd1bcfa8d53ad2e6a46c4caf63386981dac6d14b50896f7bc4264bd6484f8"}
Jul 22 03:13:50.084451 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:50.084326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2" event={"ID":"e0f95d49-a2f8-4333-b17b-83a4ef01ee05","Type":"ContainerStarted","Data":"f93825ddffe3c20eff74f8ecaf049de3da3b350fa6f5718686d792b69ca46801"}
Jul 22 03:13:50.099904 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:50.099852    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2" podStartSLOduration=1.099834427 podStartE2EDuration="1.099834427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:13:50.09949155 +0000 UTC m=+2719.780224596" watchObservedRunningTime="2026-07-22 03:13:50.099834427 +0000 UTC m=+2719.780567475"
Jul 22 03:13:51.089905 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:51.089867    2583 generic.go:356] "Generic (PLEG): container finished" podID="e0f95d49-a2f8-4333-b17b-83a4ef01ee05" containerID="cd1fd1bcfa8d53ad2e6a46c4caf63386981dac6d14b50896f7bc4264bd6484f8" exitCode=0
Jul 22 03:13:51.090314 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:51.089968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2" event={"ID":"e0f95d49-a2f8-4333-b17b-83a4ef01ee05","Type":"ContainerDied","Data":"cd1fd1bcfa8d53ad2e6a46c4caf63386981dac6d14b50896f7bc4264bd6484f8"}
Jul 22 03:13:52.227942 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:52.227913    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"
Jul 22 03:13:52.272375 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:52.272341    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f95d49-a2f8-4333-b17b-83a4ef01ee05-maas-api-service-ca\") pod \"e0f95d49-a2f8-4333-b17b-83a4ef01ee05\" (UID: \"e0f95d49-a2f8-4333-b17b-83a4ef01ee05\") "
Jul 22 03:13:52.272738 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:52.272710    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f95d49-a2f8-4333-b17b-83a4ef01ee05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0f95d49-a2f8-4333-b17b-83a4ef01ee05" (UID: "e0f95d49-a2f8-4333-b17b-83a4ef01ee05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:13:52.373351 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:52.373272    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0f95d49-a2f8-4333-b17b-83a4ef01ee05-maas-api-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:13:53.100596 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:53.100562    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2" event={"ID":"e0f95d49-a2f8-4333-b17b-83a4ef01ee05","Type":"ContainerDied","Data":"f93825ddffe3c20eff74f8ecaf049de3da3b350fa6f5718686d792b69ca46801"}
Jul 22 03:13:53.100765 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:53.100605    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"
Jul 22 03:13:53.100765 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:53.100611    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f93825ddffe3c20eff74f8ecaf049de3da3b350fa6f5718686d792b69ca46801"
Jul 22 03:13:59.945844 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:59.945799    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"]
Jul 22 03:13:59.946351 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:13:59.946060    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58" podUID="d734b7ff-3e92-4c19-953f-9839786ceeb7" containerName="maas-api" containerID="cri-o://1b6478c184badeb16831b8932700e4503806bc62c64a5c131b8559fbc467aab1" gracePeriod=30
Jul 22 03:14:00.134626 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.134574    2583 generic.go:356] "Generic (PLEG): container finished" podID="d734b7ff-3e92-4c19-953f-9839786ceeb7" containerID="1b6478c184badeb16831b8932700e4503806bc62c64a5c131b8559fbc467aab1" exitCode=0
Jul 22 03:14:00.134808 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.134655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58" event={"ID":"d734b7ff-3e92-4c19-953f-9839786ceeb7","Type":"ContainerDied","Data":"1b6478c184badeb16831b8932700e4503806bc62c64a5c131b8559fbc467aab1"}
Jul 22 03:14:00.187848 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.187826    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:14:00.243273 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.243243    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d734b7ff-3e92-4c19-953f-9839786ceeb7-maas-api-tls\") pod \"d734b7ff-3e92-4c19-953f-9839786ceeb7\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") "
Jul 22 03:14:00.243477 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.243317    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d734b7ff-3e92-4c19-953f-9839786ceeb7-openshift-service-ca\") pod \"d734b7ff-3e92-4c19-953f-9839786ceeb7\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") "
Jul 22 03:14:00.243477 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.243356    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxchh\" (UniqueName: \"kubernetes.io/projected/d734b7ff-3e92-4c19-953f-9839786ceeb7-kube-api-access-pxchh\") pod \"d734b7ff-3e92-4c19-953f-9839786ceeb7\" (UID: \"d734b7ff-3e92-4c19-953f-9839786ceeb7\") "
Jul 22 03:14:00.243675 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.243651    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d734b7ff-3e92-4c19-953f-9839786ceeb7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d734b7ff-3e92-4c19-953f-9839786ceeb7" (UID: "d734b7ff-3e92-4c19-953f-9839786ceeb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 03:14:00.245614 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.245588    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d734b7ff-3e92-4c19-953f-9839786ceeb7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d734b7ff-3e92-4c19-953f-9839786ceeb7" (UID: "d734b7ff-3e92-4c19-953f-9839786ceeb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 03:14:00.245723 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.245672    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d734b7ff-3e92-4c19-953f-9839786ceeb7-kube-api-access-pxchh" (OuterVolumeSpecName: "kube-api-access-pxchh") pod "d734b7ff-3e92-4c19-953f-9839786ceeb7" (UID: "d734b7ff-3e92-4c19-953f-9839786ceeb7"). InnerVolumeSpecName "kube-api-access-pxchh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 03:14:00.344783 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.344751    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d734b7ff-3e92-4c19-953f-9839786ceeb7-maas-api-tls\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:14:00.344783 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.344780    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d734b7ff-3e92-4c19-953f-9839786ceeb7-openshift-service-ca\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:14:00.344783 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:00.344790    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxchh\" (UniqueName: \"kubernetes.io/projected/d734b7ff-3e92-4c19-953f-9839786ceeb7-kube-api-access-pxchh\") on node \"ip-10-0-133-30.ec2.internal\" DevicePath \"\""
Jul 22 03:14:01.140099 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:01.139982    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"
Jul 22 03:14:01.140099 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:01.140015    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58" event={"ID":"d734b7ff-3e92-4c19-953f-9839786ceeb7","Type":"ContainerDied","Data":"742acb20c79b98842bf702f12f458882f7070c4dbf5ccb12048586ce3416d8f3"}
Jul 22 03:14:01.140584 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:01.140117    2583 scope.go:117] "RemoveContainer" containerID="1b6478c184badeb16831b8932700e4503806bc62c64a5c131b8559fbc467aab1"
Jul 22 03:14:01.164402 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:01.164369    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"]
Jul 22 03:14:01.168918 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:01.168890    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e6d029-7684665ccb-84m58"]
Jul 22 03:14:02.868151 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:02.868118    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d734b7ff-3e92-4c19-953f-9839786ceeb7" path="/var/lib/kubelet/pods/d734b7ff-3e92-4c19-953f-9839786ceeb7/volumes"
Jul 22 03:14:04.432566 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:04.432529    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"]
Jul 22 03:14:04.435467 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:04.435441    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-e6d029-lh4h2"]
Jul 22 03:14:04.868773 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:04.868737    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f95d49-a2f8-4333-b17b-83a4ef01ee05" path="/var/lib/kubelet/pods/e0f95d49-a2f8-4333-b17b-83a4ef01ee05/volumes"
Jul 22 03:14:11.955919 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:11.955867    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67d8987c54-zl6mw_5ef93db4-29c6-454c-8641-83f0d1fef925/authorino/0.log"
Jul 22 03:14:17.657519 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:17.657487    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l5pnp_cd6df209-b6e5-4aa0-aef3-543b387a741e/manager/0.log"
Jul 22 03:14:18.417215 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:18.417187    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c79b8765-q2zdv_659b312c-abf5-4824-90e0-3786f049a3a8/copy-manifests/0.log"
Jul 22 03:14:18.424881 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:18.424849    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c79b8765-q2zdv_659b312c-abf5-4824-90e0-3786f049a3a8/manager/0.log"
Jul 22 03:14:18.778363 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:18.778333    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7c4dcdb54c-k42cz_3ab2a44f-66ee-424c-a923-1ad334f56918/maas-api/0.log"
Jul 22 03:14:19.121024 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:19.120944    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-nkhww_30019bad-195a-4e7b-8406-45f3772c4e20/postgres/0.log"
Jul 22 03:14:20.470010 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:20.469983    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67d8987c54-zl6mw_5ef93db4-29c6-454c-8641-83f0d1fef925/authorino/0.log"
Jul 22 03:14:20.840813 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:20.840787    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-52ptn_db88cc7f-7510-4c8c-96b1-2175a4e706b9/kuadrant-console-plugin/0.log"
Jul 22 03:14:22.056784 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:22.056758    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fd555b8c5-q4pjh_a460d2a8-9522-4b15-9997-064d5005bf0c/kube-auth-proxy/0.log"
Jul 22 03:14:22.431017 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:22.430941    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-9fdns_82bb7506-cf1e-4e7c-b747-abc86314f7ad/payload-processing/0.log"
Jul 22 03:14:23.378803 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:23.378771    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zh4vf_20f83412-e6e4-4341-8dd5-666c863548bb/main/0.log"
Jul 22 03:14:23.494791 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:23.494746    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-h9xrq_90d56769-7809-46f2-bbad-5cbb65e85fee/main/0.log"
Jul 22 03:14:29.410436 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.410399    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gt8kb/must-gather-v8m9s"]
Jul 22 03:14:29.410816 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.410734    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d734b7ff-3e92-4c19-953f-9839786ceeb7" containerName="maas-api"
Jul 22 03:14:29.410816 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.410749    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d734b7ff-3e92-4c19-953f-9839786ceeb7" containerName="maas-api"
Jul 22 03:14:29.410816 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.410760    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f95d49-a2f8-4333-b17b-83a4ef01ee05" containerName="revoke-keys"
Jul 22 03:14:29.410816 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.410765    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f95d49-a2f8-4333-b17b-83a4ef01ee05" containerName="revoke-keys"
Jul 22 03:14:29.410951 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.410823    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d734b7ff-3e92-4c19-953f-9839786ceeb7" containerName="maas-api"
Jul 22 03:14:29.410951 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.410833    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f95d49-a2f8-4333-b17b-83a4ef01ee05" containerName="revoke-keys"
Jul 22 03:14:29.418825 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.418796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gt8kb/must-gather-v8m9s"
Jul 22 03:14:29.421414 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.421390    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gt8kb\"/\"kube-root-ca.crt\""
Jul 22 03:14:29.422191 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.422172    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gt8kb\"/\"openshift-service-ca.crt\""
Jul 22 03:14:29.422289 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.422179    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gt8kb\"/\"default-dockercfg-sxttz\""
Jul 22 03:14:29.425011 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.424988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gt8kb/must-gather-v8m9s"]
Jul 22 03:14:29.499205 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.499177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtng\" (UniqueName: \"kubernetes.io/projected/dea25155-47a2-460b-ae79-0c29a21c6160-kube-api-access-kxtng\") pod \"must-gather-v8m9s\" (UID: \"dea25155-47a2-460b-ae79-0c29a21c6160\") " pod="openshift-must-gather-gt8kb/must-gather-v8m9s"
Jul 22 03:14:29.499384 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.499237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dea25155-47a2-460b-ae79-0c29a21c6160-must-gather-output\") pod \"must-gather-v8m9s\" (UID: \"dea25155-47a2-460b-ae79-0c29a21c6160\") " pod="openshift-must-gather-gt8kb/must-gather-v8m9s"
Jul 22 03:14:29.600186 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.600140    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxtng\" (UniqueName: \"kubernetes.io/projected/dea25155-47a2-460b-ae79-0c29a21c6160-kube-api-access-kxtng\") pod \"must-gather-v8m9s\" (UID: \"dea25155-47a2-460b-ae79-0c29a21c6160\") " pod="openshift-must-gather-gt8kb/must-gather-v8m9s"
Jul 22 03:14:29.600378 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.600230    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dea25155-47a2-460b-ae79-0c29a21c6160-must-gather-output\") pod \"must-gather-v8m9s\" (UID: \"dea25155-47a2-460b-ae79-0c29a21c6160\") " pod="openshift-must-gather-gt8kb/must-gather-v8m9s"
Jul 22 03:14:29.600537 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.600517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/dea25155-47a2-460b-ae79-0c29a21c6160-must-gather-output\") pod \"must-gather-v8m9s\" (UID: \"dea25155-47a2-460b-ae79-0c29a21c6160\") " pod="openshift-must-gather-gt8kb/must-gather-v8m9s"
Jul 22 03:14:29.608480 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.608456    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxtng\" (UniqueName: \"kubernetes.io/projected/dea25155-47a2-460b-ae79-0c29a21c6160-kube-api-access-kxtng\") pod \"must-gather-v8m9s\" (UID: \"dea25155-47a2-460b-ae79-0c29a21c6160\") " pod="openshift-must-gather-gt8kb/must-gather-v8m9s"
Jul 22 03:14:29.728124 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.728098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gt8kb/must-gather-v8m9s"
Jul 22 03:14:29.864189 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:29.864164    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gt8kb/must-gather-v8m9s"]
Jul 22 03:14:29.866022 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:14:29.865993    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddea25155_47a2_460b_ae79_0c29a21c6160.slice/crio-8a37984f82fd45d929ff5d9f80733ea06ec12925647ef3ec5b92b1755c8c155f WatchSource:0}: Error finding container 8a37984f82fd45d929ff5d9f80733ea06ec12925647ef3ec5b92b1755c8c155f: Status 404 returned error can't find the container with id 8a37984f82fd45d929ff5d9f80733ea06ec12925647ef3ec5b92b1755c8c155f
Jul 22 03:14:30.278622 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:30.278581    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gt8kb/must-gather-v8m9s" event={"ID":"dea25155-47a2-460b-ae79-0c29a21c6160","Type":"ContainerStarted","Data":"8a37984f82fd45d929ff5d9f80733ea06ec12925647ef3ec5b92b1755c8c155f"}
Jul 22 03:14:31.285486 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:31.285448    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gt8kb/must-gather-v8m9s" event={"ID":"dea25155-47a2-460b-ae79-0c29a21c6160","Type":"ContainerStarted","Data":"dbc0ab8ea8c8dde86c23da39dc77212b17a1057b5157943b24f6155cbda36a8e"}
Jul 22 03:14:31.285486 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:31.285499    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gt8kb/must-gather-v8m9s" event={"ID":"dea25155-47a2-460b-ae79-0c29a21c6160","Type":"ContainerStarted","Data":"307bd3ec8c63ee7b3dd9e3bb7d0a4462f6f3703771a5fae7ae341580a69f1e3e"}
Jul 22 03:14:31.303091 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:31.303013    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gt8kb/must-gather-v8m9s" podStartSLOduration=1.40675179 podStartE2EDuration="2.302989241s" totalImagesPullingTime="896.237451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 03:14:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:14:31.300640807 +0000 UTC m=+2760.981373852" watchObservedRunningTime="2026-07-22 03:14:31.302989241 +0000 UTC m=+2760.983722288"
Jul 22 03:14:32.348337 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:32.348310    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6grr5_cdcb05cc-be7e-40e0-943d-e84bc42a16dc/global-pull-secret-syncer/0.log"
Jul 22 03:14:32.563985 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:32.563957    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qpnz2_e2e694d6-f4a6-48bf-a27c-219fcd70e25c/konnectivity-agent/0.log"
Jul 22 03:14:32.674264 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:32.674179    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-30.ec2.internal_bc224443957a17fe5223c8975763ab96/haproxy/0.log"
Jul 22 03:14:37.905473 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:37.905357    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67d8987c54-zl6mw_5ef93db4-29c6-454c-8641-83f0d1fef925/authorino/0.log"
Jul 22 03:14:38.003324 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:38.003291    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-52ptn_db88cc7f-7510-4c8c-96b1-2175a4e706b9/kuadrant-console-plugin/0.log"
Jul 22 03:14:40.196963 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.196915    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4af2adf2-6cfd-458a-b722-e9fc30649923/alertmanager/0.log"
Jul 22 03:14:40.225206 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.225176    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4af2adf2-6cfd-458a-b722-e9fc30649923/config-reloader/0.log"
Jul 22 03:14:40.250954 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.250929    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4af2adf2-6cfd-458a-b722-e9fc30649923/kube-rbac-proxy-web/0.log"
Jul 22 03:14:40.280547 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.280462    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4af2adf2-6cfd-458a-b722-e9fc30649923/kube-rbac-proxy/0.log"
Jul 22 03:14:40.306450 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.306424    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4af2adf2-6cfd-458a-b722-e9fc30649923/kube-rbac-proxy-metric/0.log"
Jul 22 03:14:40.336841 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.336800    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4af2adf2-6cfd-458a-b722-e9fc30649923/prom-label-proxy/0.log"
Jul 22 03:14:40.362929 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.362907    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4af2adf2-6cfd-458a-b722-e9fc30649923/init-config-reloader/0.log"
Jul 22 03:14:40.565218 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.564998    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-mhm8p_d17ff1de-43a2-4436-bc0f-3a2cde1dd356/monitoring-plugin/0.log"
Jul 22 03:14:40.704153 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.704118    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c62tt_8c5ac591-dd52-4dc6-b246-b71761defe9f/node-exporter/0.log"
Jul 22 03:14:40.729668 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.729609    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c62tt_8c5ac591-dd52-4dc6-b246-b71761defe9f/kube-rbac-proxy/0.log"
Jul 22 03:14:40.755425 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.755402    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c62tt_8c5ac591-dd52-4dc6-b246-b71761defe9f/init-textfile/0.log"
Jul 22 03:14:40.990157 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:40.990089    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10/prometheus/0.log"
Jul 22 03:14:41.019282 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:41.019252    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10/config-reloader/0.log"
Jul 22 03:14:41.047269 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:41.047239    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10/thanos-sidecar/0.log"
Jul 22 03:14:41.083802 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:41.083772    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10/kube-rbac-proxy-web/0.log"
Jul 22 03:14:41.112165 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:41.112139    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10/kube-rbac-proxy/0.log"
Jul 22 03:14:41.140456 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:41.140413    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10/kube-rbac-proxy-thanos/0.log"
Jul 22 03:14:41.177269 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:41.177235    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b79cfd6e-8b5b-442e-baf2-4eebc5b7ee10/init-config-reloader/0.log"
Jul 22 03:14:41.230178 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:41.230089    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8jg7g_29ecbb10-336d-4848-ad50-f7e35431b349/prometheus-operator/0.log"
Jul 22 03:14:41.253358 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:41.253275    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8jg7g_29ecbb10-336d-4848-ad50-f7e35431b349/kube-rbac-proxy/0.log"
Jul 22 03:14:42.770570 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.770529    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"]
Jul 22 03:14:42.778103 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.778073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.782274 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.781764    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"]
Jul 22 03:14:42.837953 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.837684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-sys\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.837953 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.837750    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk5cr\" (UniqueName: \"kubernetes.io/projected/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-kube-api-access-lk5cr\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.837953 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.837811    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-lib-modules\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.837953 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.837843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-proc\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.837953 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.837872    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-podres\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.939304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-lib-modules\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.939358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-proc\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.939392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-podres\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.939445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-sys\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.939499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk5cr\" (UniqueName: \"kubernetes.io/projected/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-kube-api-access-lk5cr\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.939980    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-lib-modules\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.940078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-proc\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.940164    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-podres\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.940244 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.940216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-sys\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:42.948439 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:42.948412    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk5cr\" (UniqueName: \"kubernetes.io/projected/c0891f06-d606-4b5f-8cb4-0005a7dfa82f-kube-api-access-lk5cr\") pod \"perf-node-gather-daemonset-rbrsm\" (UID: \"c0891f06-d606-4b5f-8cb4-0005a7dfa82f\") " pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:43.098451 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:43.098165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:43.290444 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:43.290337    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"]
Jul 22 03:14:43.294930 ip-10-0-133-30 kubenswrapper[2583]: W0722 03:14:43.294895    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc0891f06_d606_4b5f_8cb4_0005a7dfa82f.slice/crio-88016dfa0a4ee41ccff3638c7e317f51d4a030b43c31570710b40a3430122875 WatchSource:0}: Error finding container 88016dfa0a4ee41ccff3638c7e317f51d4a030b43c31570710b40a3430122875: Status 404 returned error can't find the container with id 88016dfa0a4ee41ccff3638c7e317f51d4a030b43c31570710b40a3430122875
Jul 22 03:14:43.372097 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:43.370341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm" event={"ID":"c0891f06-d606-4b5f-8cb4-0005a7dfa82f","Type":"ContainerStarted","Data":"88016dfa0a4ee41ccff3638c7e317f51d4a030b43c31570710b40a3430122875"}
Jul 22 03:14:44.377253 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:44.377207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm" event={"ID":"c0891f06-d606-4b5f-8cb4-0005a7dfa82f","Type":"ContainerStarted","Data":"4c6ab8ba740f4783530e995602cc1c0f39e231efacc4424a5fbf6beb49d3cce4"}
Jul 22 03:14:44.378076 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:44.378018    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:44.393434 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:44.393382    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm" podStartSLOduration=2.393364003 podStartE2EDuration="2.393364003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 03:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 03:14:44.392477287 +0000 UTC m=+2774.073210333" watchObservedRunningTime="2026-07-22 03:14:44.393364003 +0000 UTC m=+2774.074097050"
Jul 22 03:14:45.945402 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:45.945372    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6fbg_8a283d73-3254-49ee-bd3d-32015aea25e0/dns/0.log"
Jul 22 03:14:45.980290 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:45.980265    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x6fbg_8a283d73-3254-49ee-bd3d-32015aea25e0/kube-rbac-proxy/0.log"
Jul 22 03:14:46.044207 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:46.044182    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t2hdl_a4e8bb86-f9d8-440a-a1eb-0f59bb913304/dns-node-resolver/0.log"
Jul 22 03:14:46.711351 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:46.711318    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w8jgr_ad9a4e2d-73bb-4e36-84f2-dba812e2812f/node-ca/0.log"
Jul 22 03:14:47.900657 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:47.900631    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fd555b8c5-q4pjh_a460d2a8-9522-4b15-9997-064d5005bf0c/kube-auth-proxy/0.log"
Jul 22 03:14:48.067407 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:48.067380    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-9fdns_82bb7506-cf1e-4e7c-b747-abc86314f7ad/payload-processing/0.log"
Jul 22 03:14:48.874395 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:48.874367    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vx8b4_1f191e60-ef7e-4fc1-93da-9081634c8ba4/serve-healthcheck-canary/0.log"
Jul 22 03:14:49.592553 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:49.592521    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njphq_eb1cb847-7b39-496c-ab68-ed6478050b8e/kube-rbac-proxy/0.log"
Jul 22 03:14:49.617234 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:49.617169    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njphq_eb1cb847-7b39-496c-ab68-ed6478050b8e/exporter/0.log"
Jul 22 03:14:49.651865 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:49.651834    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-njphq_eb1cb847-7b39-496c-ab68-ed6478050b8e/extractor/0.log"
Jul 22 03:14:51.398999 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:51.398914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gt8kb/perf-node-gather-daemonset-rbrsm"
Jul 22 03:14:52.072430 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:52.072399    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l5pnp_cd6df209-b6e5-4aa0-aef3-543b387a741e/manager/0.log"
Jul 22 03:14:52.601757 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:52.601728    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c79b8765-q2zdv_659b312c-abf5-4824-90e0-3786f049a3a8/manager/0.log"
Jul 22 03:14:52.629935 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:52.629909    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86c79b8765-q2zdv_659b312c-abf5-4824-90e0-3786f049a3a8/copy-manifests/0.log"
Jul 22 03:14:53.596303 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:53.596199    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fd555b8c5-q4pjh_a460d2a8-9522-4b15-9997-064d5005bf0c/kube-auth-proxy/0.log"
Jul 22 03:14:53.730613 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:53.730581    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-9fdns_82bb7506-cf1e-4e7c-b747-abc86314f7ad/payload-processing/0.log"
Jul 22 03:14:54.135813 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:54.135720    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57d76bdcd7-qr4bm_629f68b7-5d4e-4bc3-b58c-a0ad98348a76/manager/0.log"
Jul 22 03:14:55.095164 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:55.095135    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fd555b8c5-q4pjh_a460d2a8-9522-4b15-9997-064d5005bf0c/kube-auth-proxy/0.log"
Jul 22 03:14:55.224084 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:14:55.224051    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-9fdns_82bb7506-cf1e-4e7c-b747-abc86314f7ad/payload-processing/0.log"
Jul 22 03:15:00.511246 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:00.511216    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7xjsv_446f6070-adf6-4559-b8e1-d0e7fc6fff57/kube-multus/0.log"
Jul 22 03:15:00.916701 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:00.916672    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zfxnf_f5aed715-01ac-4ee0-a911-aae835161a19/kube-multus-additional-cni-plugins/0.log"
Jul 22 03:15:00.941600 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:00.941575    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zfxnf_f5aed715-01ac-4ee0-a911-aae835161a19/egress-router-binary-copy/0.log"
Jul 22 03:15:00.963348 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:00.963320    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zfxnf_f5aed715-01ac-4ee0-a911-aae835161a19/cni-plugins/0.log"
Jul 22 03:15:00.986798 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:00.986765    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zfxnf_f5aed715-01ac-4ee0-a911-aae835161a19/bond-cni-plugin/0.log"
Jul 22 03:15:01.010004 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:01.009970    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zfxnf_f5aed715-01ac-4ee0-a911-aae835161a19/routeoverride-cni/0.log"
Jul 22 03:15:01.032557 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:01.032527    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zfxnf_f5aed715-01ac-4ee0-a911-aae835161a19/whereabouts-cni-bincopy/0.log"
Jul 22 03:15:01.057043 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:01.057006    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zfxnf_f5aed715-01ac-4ee0-a911-aae835161a19/whereabouts-cni/0.log"
Jul 22 03:15:01.233582 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:01.233554    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5t4kv_b1b9f796-93fd-4c18-a461-4894a9ce9d0c/network-metrics-daemon/0.log"
Jul 22 03:15:01.258304 ip-10-0-133-30 kubenswrapper[2583]: I0722 03:15:01.258275    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5t4kv_b1b9f796-93fd-4c18-a461-4894a9ce9d0c/kube-rbac-proxy/0.log"