Jul 28 01:25:39.323022 ip-10-0-142-30 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 01:25:39.678130 ip-10-0-142-30 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:39.679440 ip-10-0-142-30 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 01:25:39.679440 ip-10-0-142-30 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:39.679440 ip-10-0-142-30 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 01:25:39.679544 ip-10-0-142-30 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:39.680253 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.680143    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 01:25:39.685555 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685541    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:39.685555 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685555    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685559    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685562    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685565    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685568    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685571    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685576    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685579    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685581    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685584    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685587    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685589    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685592    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685594    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685597    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685600    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685602    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685605    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685607    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:39.685621 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685611    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685613    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685615    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685618    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685621    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685624    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685627    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685629    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685632    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685635    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685638    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685640    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685643    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685646    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685648    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685651    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685653    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685656    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685661    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:39.686094 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685670    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685674    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685676    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685679    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685682    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685684    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685687    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685689    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685692    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685694    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685697    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685699    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685702    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685705    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685707    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685710    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685714    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685717    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685720    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685723    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:39.686554 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685726    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685729    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685731    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685734    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685736    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685740    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685744    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685746    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685749    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685763    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685766    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685769    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685778    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685781    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685783    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685786    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685789    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685791    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685794    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685796    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:39.687059 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685799    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685803    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685805    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685808    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685810    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685812    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685815    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685819    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685822    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685825    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685827    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685830    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685833    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685835    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685839    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685843    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685847    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685850    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685852    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.685855    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:39.687542 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686540    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686546    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686550    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686552    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686555    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686558    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686561    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686564    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686567    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686569    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686572    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686574    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686577    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686580    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686582    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686585    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686587    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686589    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686592    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686595    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686598    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:39.688052 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686602    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686605    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686607    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686610    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686612    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686615    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686617    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686620    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686624    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686628    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686630    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686633    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686636    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686639    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686642    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686645    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686648    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686651    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686653    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:39.688575 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686656    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686660    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686664    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686666    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686669    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686671    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686675    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686678    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686680    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686683    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686685    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686688    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686691    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686693    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686696    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686699    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686701    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686703    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686706    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:39.689049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686708    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686711    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686713    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686716    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686718    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686720    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686723    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686725    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686728    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686730    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686733    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686735    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686738    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686740    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686743    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686745    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686748    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686750    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686766    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686769    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:39.689508 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686772    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686774    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686777    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686780    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686783    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686786    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686789    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686792    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686794    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686797    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686799    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686803    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686805    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686809    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686811    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686814    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686818    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686821    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686824    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:39.690015 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.686827    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687175    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687179    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687182    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687185    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687189    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687192    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687196    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687200    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687203    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687206    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687209    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687211    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687216    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687220    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687222    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687225    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687228    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687231    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687233    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:39.690507 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687236    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687238    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687241    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687243    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687246    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687248    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687251    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687253    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687256    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687259    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687262    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687265    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687267    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687269    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687272    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687274    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687277    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687279    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687282    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687285    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:39.691014 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687287    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687290    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687292    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687295    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687297    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687300    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687304    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687308    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687310    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687314    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687316    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687319    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687322    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687325    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687327    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687329    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687332    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687336    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687338    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:39.691495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687341    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687343    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687346    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687348    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687351    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687353    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687355    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687358    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687360    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687363    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687366    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687368    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687371    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687374    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687376    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687379    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687381    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687383    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687386    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687389    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:39.692049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687391    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687394    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687397    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687399    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687402    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687404    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687407    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687409    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687412    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687414    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687417    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687419    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687422    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687424    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687427    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687429    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687431    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687434    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687436    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:39.692537 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687439    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.687441    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687506    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687513    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687518    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687523    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687527    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687531    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687535    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687538    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687542    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687545    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687549    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687552    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687556    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687560    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687562    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687566    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687569    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687572    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687576    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687579    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687582    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687585    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 01:25:39.693019 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687589    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687592    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687595    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687598    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687602    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687604    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687607    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687610    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687613    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687616    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687620    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687623    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687626    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687629    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687632    2579 flags.go:64] FLAG: --enable-server="true"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687635    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687639    2579 flags.go:64] FLAG: --event-burst="100"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687643    2579 flags.go:64] FLAG: --event-qps="50"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687646    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687651    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687654    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687658    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687661    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687664    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687667    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 28 01:25:39.693625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687670    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687673    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687676    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687679    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687682    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687684    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687687    2579 flags.go:64] FLAG: --feature-gates=""
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687693    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687696    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687699    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687703    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687706    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687709    2579 flags.go:64] FLAG: --help="false"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687712    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-142-30.ec2.internal"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687715    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687718    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687721    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687724    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687727    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687730    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687733    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687736    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687739    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 01:25:39.694258 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687742    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687745    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687748    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687770    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687773    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687776    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687781    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687784    2579 flags.go:64] FLAG: --lock-file=""
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687787    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687790    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687793    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687799    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687802    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687804    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687807    2579 flags.go:64] FLAG: --logging-format="text"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687810    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687814    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687817    2579 flags.go:64] FLAG: --manifest-url=""
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687820    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687824    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687827    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687831    2579 flags.go:64] FLAG: --max-pods="110"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687834    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687837    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687840    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 01:25:39.694863 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687843    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687846    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687848    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687851    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687858    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687861    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687864    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687867    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687870    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687876    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687878    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687883    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687886    2579 flags.go:64] FLAG: --port="10250"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687890    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687894    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cb142cba43b23598"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687897    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687899    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687902    2579 flags.go:64] FLAG: --register-node="true"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687905    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687909    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687912    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687914    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687917    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687920    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687924    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 01:25:39.695489 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687927    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687929    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687932    2579 flags.go:64] FLAG: --runonce="false"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687935    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687938    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687941    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687944    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687947    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687950    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687953    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687956    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687959    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687962    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687965    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687968    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687971    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687977    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687980    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687988    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687991    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687995    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.687999    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688002    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688005    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688008    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 01:25:39.696194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688011    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688014    2579 flags.go:64] FLAG: --v="2"
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688018    2579 flags.go:64] FLAG: --version="false"
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688022    2579 flags.go:64] FLAG: --vmodule=""
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688025    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688028    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688138    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688142    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688146    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688148    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688151    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688154    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688157    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688159    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688162    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688164    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688167    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688171    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688174    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688177    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688180    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:39.696858 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688184    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688186    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688189    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688192    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688197    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688200    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688204    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688206    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688209    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688212    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688215    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688217    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688221    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688224    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688227    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688229    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688232    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688235    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688237    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688239    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:39.697383 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688242    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688245    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688248    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688250    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688253    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688255    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688258    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688260    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688263    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688265    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688268    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688271    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688273    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688275    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688278    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688280    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688285    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688287    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688292    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688294    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:39.697915 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688297    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688299    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688302    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688304    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688307    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688309    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688312    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688315    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688317    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688319    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688322    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688325    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688327    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688330    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688332    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688336    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688339    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688342    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688344    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688347    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:39.698401 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688349    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688352    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688355    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688357    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688359    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688362    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688364    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688367    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688371    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688374    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688379    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688381    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688384    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688387    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688389    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688392    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688395    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688397    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688400    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:39.698973 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688402    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688405    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688407    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688410    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.688413    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.688984    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.695438    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.695453    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.696063    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.696070    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696128    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696133    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696137    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696141    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696144    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:39.699567 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696146    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696149    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696152    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696155    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696159    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696163    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696166    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696170    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696173    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696176    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696179    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696183    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696185    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696188    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696192    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696196    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696200    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696204    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696206    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:39.700028 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696209    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696212    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696215    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696218    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696221    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696223    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696227    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696229    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696232    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696235    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696237    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696240    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696242    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696245    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696248    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696251    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696253    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696256    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696258    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696260    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:39.700479 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696263    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696266    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696269    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696272    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696275    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696278    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696281    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696283    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696286    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696288    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696291    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696293    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696296    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696298    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696301    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696303    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696306    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696308    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696311    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696314    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:39.700975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696316    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696319    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696321    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696323    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696326    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696328    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696331    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696333    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696337    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696340    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696344    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696347    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696350    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696353    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696361    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696364    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696367    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696369    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696372    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:39.701459 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696374    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696377    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696380    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696382    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696385    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696387    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696390    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696393    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696395    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696398    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696400    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696403    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696406    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696408    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696411    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:39.701949 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696413    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.696418    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696530    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696535    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696538    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696542    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696544    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696547    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696550    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696554    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696556    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696559    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696561    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696566    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696568    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:39.702305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696571    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696574    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696577    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696579    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696581    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696584    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696586    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696590    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696594    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696597    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696599    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696602    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696604    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696607    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696610    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696612    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696615    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696618    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696620    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696623    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:39.702674 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696626    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696628    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696630    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696633    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696636    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696638    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696641    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696643    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696646    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696648    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696651    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696654    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696656    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696659    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696661    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696664    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696666    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696669    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696673    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:39.703173 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696676    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696679    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696682    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696685    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696687    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696690    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696692    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696695    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696697    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696700    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696702    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696705    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696707    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696710    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696712    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696715    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696717    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696720    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696722    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696725    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:39.703625 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696728    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696730    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696733    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696736    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696739    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696742    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696744    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696747    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696750    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696764    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696767    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696770    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696773    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696775    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696778    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696780    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696783    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696785    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696788    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:39.704196 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696791    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696793    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696796    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696798    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696801    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696803    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696806    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:39.696808    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.696813    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.696920    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.700283    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.701495    2579 server.go:1019] "Starting client certificate rotation"
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.701591    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:39.704669 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.701629    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:39.705008 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.703735    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:39.707347 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.707328    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:39.719830 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.719815    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 28 01:25:39.723555 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.723530    2579 log.go:25] "Validated CRI v1 image API"
Jul 28 01:25:39.724600 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.724585    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 01:25:39.726793 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.726775    2579 fs.go:135] Filesystem UUIDs: map[329a1a80-cac5-4c44-992a-915ad708cc64:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 eeb5ed8f-ac3f-4873-81bc-a4dca1c11512:/dev/nvme0n1p4]
Jul 28 01:25:39.726850 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.726793    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 01:25:39.729168 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.729154    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:39.732623 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.732516    2579 manager.go:216] Machine: {Timestamp:2026-07-28 01:25:39.731368568 +0000 UTC m=+0.305585076 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098580 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2653715733a19ff735b1b39011b52e SystemUUID:ec265371-5733-a19f-f735-b1b39011b52e BootID:c6fec19d-72e0-4304-b15d-dc7529b710b0 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:85:df:2a:85:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:85:df:2a:85:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:33:8c:19:43:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 01:25:39.732623 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.732616    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 01:25:39.732768 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.732691    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 01:25:39.735527 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.735502    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 01:25:39.735655 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.735530    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 01:25:39.736189 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.736177    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 01:25:39.736226 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.736193    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 01:25:39.736226 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.736200    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:39.737008 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.736999    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:39.737841 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.737831    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 01:25:39.739608 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.739598    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:39.739711 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.739701    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 01:25:39.739772 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.739731    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 01:25:39.739772 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.739741    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 01:25:39.739772 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.739766    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 01:25:39.739886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.739776    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 28 01:25:39.739886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.739786    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 01:25:39.740143 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.740130    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:39.740208 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.740148    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:39.743261 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.743210    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 01:25:39.745135 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.745118    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 01:25:39.745210 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.745148    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 01:25:39.746895 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746882    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 01:25:39.746895 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746898    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746905    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746910    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746920    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746925    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746932    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746937    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746944    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746949    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746957    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 01:25:39.746982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.746963    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 01:25:39.747694 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.747685    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 01:25:39.747694 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.747693    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 01:25:39.750728 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.750710    2579 server.go:1268] "Started kubelet"
Jul 28 01:25:39.750822 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.750774    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 01:25:39.750953 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.750896    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 01:25:39.751021 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.750978    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 01:25:39.751614 ip-10-0-142-30 systemd[1]: Started Kubernetes Kubelet.
Jul 28 01:25:39.752530 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.752514    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 01:25:39.752594 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.752511    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 01:25:39.752812 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.752680    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 01:25:39.752812 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.752679    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 01:25:39.753534 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.753520    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 01:25:39.755121 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.755103    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 01:25:39.756988 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.756974    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5g6k5"
Jul 28 01:25:39.758296 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.757565    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-30.ec2.internal.18c64fb4af2d7238  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-30.ec2.internal,UID:ip-10-0-142-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-30.ec2.internal,},FirstTimestamp:2026-07-28 01:25:39.750679096 +0000 UTC m=+0.324895610,LastTimestamp:2026-07-28 01:25:39.750679096 +0000 UTC m=+0.324895610,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-30.ec2.internal,}"
Jul 28 01:25:39.760378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.760361    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 01:25:39.760378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.760368    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:39.760508 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.760468    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:39.760910 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.760891    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 01:25:39.760910 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.760892    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 01:25:39.761051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.760929    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 01:25:39.761099 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761069    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 01:25:39.761099 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761078    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 01:25:39.761237 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761222    2579 factory.go:55] Registering systemd factory
Jul 28 01:25:39.761285 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761239    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 28 01:25:39.761469 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761453    2579 factory.go:153] Registering CRI-O factory
Jul 28 01:25:39.761469 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761469    2579 factory.go:223] Registration of the crio container factory successfully
Jul 28 01:25:39.761592 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761514    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 01:25:39.761592 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761532    2579 factory.go:103] Registering Raw factory
Jul 28 01:25:39.761592 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.761544    2579 manager.go:1196] Started watching for new ooms in manager
Jul 28 01:25:39.762069 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.762057    2579 manager.go:349] Starting recovery of all containers
Jul 28 01:25:39.764923 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.764896    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 01:25:39.765038 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.765003    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 01:25:39.765209 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.765191    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5g6k5"
Jul 28 01:25:39.773809 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.773676    2579 manager.go:354] Recovery completed
Jul 28 01:25:39.778505 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.778491    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:39.780188 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.780174    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:39.780252 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.780202    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:39.780252 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.780212    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:39.780642 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.780628    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 01:25:39.780642 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.780642    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 01:25:39.780725 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.780652    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:39.780725 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.780658    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:39.781362 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.781347    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:39.781418 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.781372    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:39.781418 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.781385    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:39.782339 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.782232    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-30.ec2.internal.18c64fb4b0efbc19  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-30.ec2.internal,UID:ip-10-0-142-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-30.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-30.ec2.internal,},FirstTimestamp:2026-07-28 01:25:39.780189209 +0000 UTC m=+0.354405716,LastTimestamp:2026-07-28 01:25:39.780189209 +0000 UTC m=+0.354405716,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-30.ec2.internal,}"
Jul 28 01:25:39.782536 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.782523    2579 policy_none.go:49] "None policy: Start"
Jul 28 01:25:39.782595 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.782545    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 01:25:39.782595 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.782559    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 01:25:39.783820 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.783806    2579 policy_none.go:47] "Start"
Jul 28 01:25:39.820490 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.820299    2579 manager.go:337] "Starting Device Plugin manager"
Jul 28 01:25:39.820598 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.820513    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 01:25:39.820598 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.820525    2579 server.go:87] "Starting device plugin registration server"
Jul 28 01:25:39.820768 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.820740    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 01:25:39.820820 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.820767    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 01:25:39.820858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.820823    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 01:25:39.820915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.820903    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 01:25:39.820915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.820915    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 01:25:39.821478 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.821448    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 01:25:39.821570 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.821521    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:39.888689 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.888647    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 01:25:39.890014 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.889877    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 01:25:39.890014 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.889920    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 01:25:39.890014 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.889948    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 01:25:39.890014 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.889989    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 01:25:39.892112 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.892091    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:39.921890 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.921856    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:39.922742 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.922725    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:39.922822 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.922769    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:39.922822 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.922784    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:39.922822 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.922811    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-30.ec2.internal"
Jul 28 01:25:39.931003 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:39.930940    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-30.ec2.internal"
Jul 28 01:25:39.931003 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:39.930963    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-30.ec2.internal\": node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:41.216608 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.216589    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:41.258357 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.258327    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal"]
Jul 28 01:25:41.258454 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.258398    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:41.259337 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.259321    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:41.259416 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.259351    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:41.259416 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.259366    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:41.260582 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.260568    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:41.260722 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.260709    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.260759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.260737    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:41.261296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.261271    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:41.261296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.261298    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:41.261471 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.261311    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:41.261471 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.261278    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:41.261471 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.261366    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:41.261471 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.261378    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:41.262519 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.262504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.262600 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.262530    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:41.263171 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.263157    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:41.263243 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.263182    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:41.263243 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.263190    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:41.290223 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.290200    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-30.ec2.internal\" not found" node="ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.294742 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.294726    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-30.ec2.internal\" not found" node="ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.317106 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.317083    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:41.330421 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.330397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1c22b9cc929456a299125322077be27-config\") pod \"kube-apiserver-proxy-ip-10-0-142-30.ec2.internal\" (UID: \"d1c22b9cc929456a299125322077be27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.330515 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.330432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28b62562efbbd88ae10aada4b859f4ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal\" (UID: \"28b62562efbbd88ae10aada4b859f4ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.330515 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.330451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b62562efbbd88ae10aada4b859f4ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal\" (UID: \"28b62562efbbd88ae10aada4b859f4ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.417787 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.417755    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:41.431532 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.431506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1c22b9cc929456a299125322077be27-config\") pod \"kube-apiserver-proxy-ip-10-0-142-30.ec2.internal\" (UID: \"d1c22b9cc929456a299125322077be27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.431532 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.431534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28b62562efbbd88ae10aada4b859f4ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal\" (UID: \"28b62562efbbd88ae10aada4b859f4ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.431690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.431552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b62562efbbd88ae10aada4b859f4ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal\" (UID: \"28b62562efbbd88ae10aada4b859f4ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.431690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.431593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28b62562efbbd88ae10aada4b859f4ab-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal\" (UID: \"28b62562efbbd88ae10aada4b859f4ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.431690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.431597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d1c22b9cc929456a299125322077be27-config\") pod \"kube-apiserver-proxy-ip-10-0-142-30.ec2.internal\" (UID: \"d1c22b9cc929456a299125322077be27\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.431690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.431638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28b62562efbbd88ae10aada4b859f4ab-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal\" (UID: \"28b62562efbbd88ae10aada4b859f4ab\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.518888 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.518795    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:41.593305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.593264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.598076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.598059    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.618892 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.618868    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:41.719340 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.719294    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:41.819894 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:41.819834    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-30.ec2.internal\" not found"
Jul 28 01:25:41.915009 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.914985    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:41.928977 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.928951    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.940806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.940781    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:41.942266 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.942251    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal"
Jul 28 01:25:41.949024 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.949011    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:41.959123 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.959106    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:41.968906 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.968885    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 01:25:41.969032 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.969010    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:41.969080 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.969011    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:41.969080 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:41.969011    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:42.008213 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.008187    2579 apiserver.go:52] "Watching apiserver"
Jul 28 01:25:42.022639 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.022612    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 01:25:42.022963 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.022937    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-jn5d4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d","openshift-cluster-node-tuning-operator/tuned-xp8sj","openshift-image-registry/node-ca-kkv68","openshift-multus/multus-additional-cni-plugins-w6knn","openshift-multus/multus-crcwz","openshift-multus/network-metrics-daemon-xskmq","kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal","openshift-dns/node-resolver-5mjjb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal","openshift-network-diagnostics/network-check-target-2x9gm","openshift-network-operator/iptables-alerter-g4vkm","openshift-ovn-kubernetes/ovnkube-node-lbrmx","kube-system/global-pull-secret-syncer-gd4x8"]
Jul 28 01:25:42.024432 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.024413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:25:42.025932 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.025912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.026509 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.026471    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 01:25:42.026574 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.026517    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qw7qp\""
Jul 28 01:25:42.026574 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.026550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 01:25:42.027179 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.027160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.028068 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.028052    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f2hn7\""
Jul 28 01:25:42.028193 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.028176    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 01:25:42.028249 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.028186    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:42.028299 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.028245    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 01:25:42.028353 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.028339    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 01:25:42.028415 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.028402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.029158 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.029141    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:42.029234 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.029187    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4v68r\""
Jul 28 01:25:42.029234 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.029142    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:42.030527 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.030513    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 01:25:42.030596 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.030514    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 01:25:42.030731 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.030717    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-djp57\""
Jul 28 01:25:42.030778 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.030764    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 01:25:42.030964 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.030944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.032205 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.032189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.032822 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.032808    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 01:25:42.032998 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.032984    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 01:25:42.032998 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.032992    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 01:25:42.033141 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.033123    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h6b6k\""
Jul 28 01:25:42.033199 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.033161    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 01:25:42.033244 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.033205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 01:25:42.033813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.033801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:42.033885 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.033869    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:42.034059 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.034045    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 01:25:42.034302 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.034290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-88t4j\""
Jul 28 01:25:42.035194 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-socket-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.035240 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-registration-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.035275 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-kubernetes\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035275 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa729e-9e90-4a5d-825c-d61951b1ca92-host\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.035348 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffaa729e-9e90-4a5d-825c-d61951b1ca92-serviceca\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.035399 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63f51959-8dc3-4814-a455-b557cfedba90-agent-certs\") pod \"konnectivity-agent-jn5d4\" (UID: \"63f51959-8dc3-4814-a455-b557cfedba90\") " pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:25:42.035399 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.035509 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.035509 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgrdg\" (UniqueName: \"kubernetes.io/projected/ad7f2e29-b4be-483b-a01e-ec496936f036-kube-api-access-tgrdg\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.035509 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-sys\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035509 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-host\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-system-cni-dir\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.035677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4lgz\" (UniqueName: \"kubernetes.io/projected/db004963-7a39-48ea-be4f-286fada36b32-kube-api-access-t4lgz\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.035677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-sys-fs\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.035677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-lib-modules\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.035677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-device-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.035677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysctl-conf\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-run\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-cnibin\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-etc-selinux\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysctl-d\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-systemd\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-var-lib-kubelet\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-tuned\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dpn\" (UniqueName: \"kubernetes.io/projected/ffaa729e-9e90-4a5d-825c-d61951b1ca92-kube-api-access-j2dpn\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035877    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.035915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63f51959-8dc3-4814-a455-b557cfedba90-konnectivity-ca\") pod \"konnectivity-agent-jn5d4\" (UID: \"63f51959-8dc3-4814-a455-b557cfedba90\") " pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:25:42.036256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-modprobe-d\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.036256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwgz\" (UniqueName: \"kubernetes.io/projected/7a8c170e-1f5f-450c-8784-db3bef797f61-kube-api-access-fbwgz\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.036256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.035972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-os-release\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.036256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.036003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.036256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.036032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.036256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.036081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysconfig\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.036256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.036105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a8c170e-1f5f-450c-8784-db3bef797f61-tmp\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.036751 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.036732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:42.036827 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.036807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:42.038545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.037691    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 01:25:42.038545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.037880    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 01:25:42.038545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.038137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gl8hm\""
Jul 28 01:25:42.038545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.038368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.039284 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.038894    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 01:20:41 +0000 UTC" deadline="2028-03-23 00:24:22.960024977 +0000 UTC"
Jul 28 01:25:42.039284 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.038966    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14494h58m40.921063104s"
Jul 28 01:25:42.039611 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.039590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.040334 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.040318    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 01:25:42.040419 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.040407    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4czj\""
Jul 28 01:25:42.040565 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.040548    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:42.040667 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.040602    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:42.041073 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.041054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.041131 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.041111    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:25:42.042522 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.042498    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 01:25:42.043104 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.043076    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-69pcn\""
Jul 28 01:25:42.043104 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.043095    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 01:25:42.043306 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.043287    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 01:25:42.043585 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.043567    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 01:25:42.043734 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.043708    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 01:25:42.043873 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.043819    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 01:25:42.045845 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.045828    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:42.064653 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.064627    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rh96m"
Jul 28 01:25:42.074446 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.074426    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rh96m"
Jul 28 01:25:42.129865 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.129843    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 01:25:42.137101 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a8c170e-1f5f-450c-8784-db3bef797f61-tmp\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.137212 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-socket-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.137212 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-registration-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.137212 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-registration-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.137305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-socket-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.137349 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-cni-multus\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.137349 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl9qb\" (UniqueName: \"kubernetes.io/projected/69084ae6-a530-4718-a6bc-2d9481b7c76c-kube-api-access-wl9qb\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.137349 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65669d60-5522-49e7-a490-1c2ac00830b1-tmp-dir\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.137349 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-slash\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.137513 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-cni-bin\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.137513 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.137513 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137420    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 01:25:42.137513 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.137638 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-sys\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.137638 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137543    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-cnibin\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.137638 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-k8s-cni-cncf-io\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.137638 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137609    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/69084ae6-a530-4718-a6bc-2d9481b7c76c-iptables-alerter-script\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.137638 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-sys\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.137638 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-env-overrides\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.137813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-dbus\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.137813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktpgz\" (UniqueName: \"kubernetes.io/projected/65669d60-5522-49e7-a490-1c2ac00830b1-kube-api-access-ktpgz\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.137813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.137813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.137813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.137813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-systemd-units\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.137813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovnkube-script-lib\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.137813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-device-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-socket-dir-parent\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-device-dir\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-ovn\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-cni-netd\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovn-node-metrics-cert\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.137950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-multus-certs\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-run-netns\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmd7z\" (UniqueName: \"kubernetes.io/projected/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-kube-api-access-xmd7z\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.138076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63f51959-8dc3-4814-a455-b557cfedba90-konnectivity-ca\") pod \"konnectivity-agent-jn5d4\" (UID: \"63f51959-8dc3-4814-a455-b557cfedba90\") " pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwgz\" (UniqueName: \"kubernetes.io/projected/7a8c170e-1f5f-450c-8784-db3bef797f61-kube-api-access-fbwgz\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-hostroot\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysconfig\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-cni-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysconfig\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-kubelet\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-cni-binary-copy\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-conf-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-etc-kubernetes\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-etc-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.138474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-kubernetes\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa729e-9e90-4a5d-825c-d61951b1ca92-host\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/63f51959-8dc3-4814-a455-b557cfedba90-konnectivity-ca\") pod \"konnectivity-agent-jn5d4\" (UID: \"63f51959-8dc3-4814-a455-b557cfedba90\") " pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffaa729e-9e90-4a5d-825c-d61951b1ca92-serviceca\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-kubernetes\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-os-release\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa729e-9e90-4a5d-825c-d61951b1ca92-host\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-daemon-config\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-kubelet\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-var-lib-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63f51959-8dc3-4814-a455-b557cfedba90-agent-certs\") pod \"konnectivity-agent-jn5d4\" (UID: \"63f51959-8dc3-4814-a455-b557cfedba90\") " pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgrdg\" (UniqueName: \"kubernetes.io/projected/ad7f2e29-b4be-483b-a01e-ec496936f036-kube-api-access-tgrdg\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-host\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-system-cni-dir\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4lgz\" (UniqueName: \"kubernetes.io/projected/db004963-7a39-48ea-be4f-286fada36b32-kube-api-access-t4lgz\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6596447-1335-4ce0-8d05-4f8ff15ea382-cni-binary-copy\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.139045 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-netns\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-host\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-sys-fs\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.138991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-system-cni-dir\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-lib-modules\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-system-cni-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-sys-fs\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-lib-modules\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzlqv\" (UniqueName: \"kubernetes.io/projected/1c6f1795-9275-4b38-ba50-f2653f2b58dc-kube-api-access-gzlqv\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65669d60-5522-49e7-a490-1c2ac00830b1-hosts-file\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-node-log\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-log-socket\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovnkube-config\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139277    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69084ae6-a530-4718-a6bc-2d9481b7c76c-host-slash\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.139762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-kubelet-config\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffaa729e-9e90-4a5d-825c-d61951b1ca92-serviceca\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysctl-conf\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-run\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-cnibin\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-cni-bin\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpdbh\" (UniqueName: \"kubernetes.io/projected/a6596447-1335-4ce0-8d05-4f8ff15ea382-kube-api-access-xpdbh\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysctl-conf\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-etc-selinux\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysctl-d\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-run\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-cnibin\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-systemd\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-var-lib-kubelet\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-tuned\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-sysctl-d\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ad7f2e29-b4be-483b-a01e-ec496936f036-etc-selinux\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-systemd\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.140554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-var-lib-kubelet\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dpn\" (UniqueName: \"kubernetes.io/projected/ffaa729e-9e90-4a5d-825c-d61951b1ca92-kube-api-access-j2dpn\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-systemd\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139814    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-modprobe-d\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-os-release\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.139946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.140016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/db004963-7a39-48ea-be4f-286fada36b32-os-release\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.140331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-modprobe-d\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.140384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.140413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/db004963-7a39-48ea-be4f-286fada36b32-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.141051 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.140896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a8c170e-1f5f-450c-8784-db3bef797f61-tmp\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.141409 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.141351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/63f51959-8dc3-4814-a455-b557cfedba90-agent-certs\") pod \"konnectivity-agent-jn5d4\" (UID: \"63f51959-8dc3-4814-a455-b557cfedba90\") " pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:25:42.141620 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.141603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a8c170e-1f5f-450c-8784-db3bef797f61-etc-tuned\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.146345 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.146284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4lgz\" (UniqueName: \"kubernetes.io/projected/db004963-7a39-48ea-be4f-286fada36b32-kube-api-access-t4lgz\") pod \"multus-additional-cni-plugins-w6knn\" (UID: \"db004963-7a39-48ea-be4f-286fada36b32\") " pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.146623 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.146601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwgz\" (UniqueName: \"kubernetes.io/projected/7a8c170e-1f5f-450c-8784-db3bef797f61-kube-api-access-fbwgz\") pod \"tuned-xp8sj\" (UID: \"7a8c170e-1f5f-450c-8784-db3bef797f61\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.147089 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.147067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgrdg\" (UniqueName: \"kubernetes.io/projected/ad7f2e29-b4be-483b-a01e-ec496936f036-kube-api-access-tgrdg\") pod \"aws-ebs-csi-driver-node-vvw9d\" (UID: \"ad7f2e29-b4be-483b-a01e-ec496936f036\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.147831 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.147815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dpn\" (UniqueName: \"kubernetes.io/projected/ffaa729e-9e90-4a5d-825c-d61951b1ca92-kube-api-access-j2dpn\") pod \"node-ca-kkv68\" (UID: \"ffaa729e-9e90-4a5d-825c-d61951b1ca92\") " pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.184649 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.184618    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b62562efbbd88ae10aada4b859f4ab.slice/crio-34cb474a55fd59d7b2d247b21fa8d548bfa4a879d1d0a6a79db14886bdc22d15 WatchSource:0}: Error finding container 34cb474a55fd59d7b2d247b21fa8d548bfa4a879d1d0a6a79db14886bdc22d15: Status 404 returned error can't find the container with id 34cb474a55fd59d7b2d247b21fa8d548bfa4a879d1d0a6a79db14886bdc22d15
Jul 28 01:25:42.185103 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.185085    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c22b9cc929456a299125322077be27.slice/crio-8d4d8f0210b380b8af7e3011250758172013a201d4eb04b4c2a9f216c46249db WatchSource:0}: Error finding container 8d4d8f0210b380b8af7e3011250758172013a201d4eb04b4c2a9f216c46249db: Status 404 returned error can't find the container with id 8d4d8f0210b380b8af7e3011250758172013a201d4eb04b4c2a9f216c46249db
Jul 28 01:25:42.190845 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.190832    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:25:42.240179 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-cni-multus\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.240179 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9qb\" (UniqueName: \"kubernetes.io/projected/69084ae6-a530-4718-a6bc-2d9481b7c76c-kube-api-access-wl9qb\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65669d60-5522-49e7-a490-1c2ac00830b1-tmp-dir\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-slash\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-cni-bin\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-cnibin\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-slash\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240324    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-cni-bin\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-cnibin\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-cni-multus\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-k8s-cni-cncf-io\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.240425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/69084ae6-a530-4718-a6bc-2d9481b7c76c-iptables-alerter-script\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-env-overrides\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-k8s-cni-cncf-io\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-dbus\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktpgz\" (UniqueName: \"kubernetes.io/projected/65669d60-5522-49e7-a490-1c2ac00830b1-kube-api-access-ktpgz\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-systemd-units\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovnkube-script-lib\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-dbus\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-socket-dir-parent\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.240730    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-ovn\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-cni-netd\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-systemd-units\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.240829    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret podName:fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:42.740791986 +0000 UTC m=+2.047280766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret") pod "global-pull-secret-syncer-gd4x8" (UID: "fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:42.240900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-cni-netd\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-ovn\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-socket-dir-parent\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/69084ae6-a530-4718-a6bc-2d9481b7c76c-iptables-alerter-script\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovn-node-metrics-cert\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-multus-certs\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-run-netns\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.240994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmd7z\" (UniqueName: \"kubernetes.io/projected/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-kube-api-access-xmd7z\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-env-overrides\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-hostroot\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-multus-certs\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-run-netns\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-cni-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241083    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65669d60-5522-49e7-a490-1c2ac00830b1-tmp-dir\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-hostroot\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-kubelet\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-cni-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-kubelet\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.241720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-conf-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-etc-kubernetes\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-etc-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-conf-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-etc-kubernetes\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-os-release\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-etc-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-daemon-config\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovnkube-script-lib\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-kubelet\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-os-release\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-var-lib-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-host-kubelet\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6596447-1335-4ce0-8d05-4f8ff15ea382-cni-binary-copy\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-netns\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-system-cni-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzlqv\" (UniqueName: \"kubernetes.io/projected/1c6f1795-9275-4b38-ba50-f2653f2b58dc-kube-api-access-gzlqv\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-run-netns\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-var-lib-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241531    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-system-cni-dir\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65669d60-5522-49e7-a490-1c2ac00830b1-hosts-file\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-node-log\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-log-socket\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovnkube-config\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-node-log\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/65669d60-5522-49e7-a490-1c2ac00830b1-hosts-file\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-log-socket\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69084ae6-a530-4718-a6bc-2d9481b7c76c-host-slash\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.241735    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-openvswitch\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.242858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6596447-1335-4ce0-8d05-4f8ff15ea382-multus-daemon-config\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.241801    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs podName:1c6f1795-9275-4b38-ba50-f2653f2b58dc nodeName:}" failed. No retries permitted until 2026-07-28 01:25:42.741764313 +0000 UTC m=+2.048253073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs") pod "network-metrics-daemon-xskmq" (UID: "1c6f1795-9275-4b38-ba50-f2653f2b58dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69084ae6-a530-4718-a6bc-2d9481b7c76c-host-slash\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-kubelet-config\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-kubelet-config\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-cni-bin\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpdbh\" (UniqueName: \"kubernetes.io/projected/a6596447-1335-4ce0-8d05-4f8ff15ea382-kube-api-access-xpdbh\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-systemd\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6596447-1335-4ce0-8d05-4f8ff15ea382-cni-binary-copy\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.241986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6596447-1335-4ce0-8d05-4f8ff15ea382-host-var-lib-cni-bin\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.242007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-run-systemd\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.242154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovnkube-config\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.243378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.243365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-ovn-node-metrics-cert\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.248916 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.248893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktpgz\" (UniqueName: \"kubernetes.io/projected/65669d60-5522-49e7-a490-1c2ac00830b1-kube-api-access-ktpgz\") pod \"node-resolver-5mjjb\" (UID: \"65669d60-5522-49e7-a490-1c2ac00830b1\") " pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.249031 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.248938    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:42.249031 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.248956    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:42.249031 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.248969    2579 projected.go:196] Error preparing data for projected volume kube-api-access-t8s2h for pod openshift-network-diagnostics/network-check-target-2x9gm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:42.249031 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.248962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9qb\" (UniqueName: \"kubernetes.io/projected/69084ae6-a530-4718-a6bc-2d9481b7c76c-kube-api-access-wl9qb\") pod \"iptables-alerter-g4vkm\" (UID: \"69084ae6-a530-4718-a6bc-2d9481b7c76c\") " pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.249031 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.249028    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h podName:f88f1a31-21c3-45c5-8cc3-f44b91806734 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:42.749013221 +0000 UTC m=+2.055501994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t8s2h" (UniqueName: "kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h") pod "network-check-target-2x9gm" (UID: "f88f1a31-21c3-45c5-8cc3-f44b91806734") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:42.249435 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.249418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmd7z\" (UniqueName: \"kubernetes.io/projected/d0ad062e-f0ff-45cc-8a4a-db2e2a833a17-kube-api-access-xmd7z\") pod \"ovnkube-node-lbrmx\" (UID: \"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.250062 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.250031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpdbh\" (UniqueName: \"kubernetes.io/projected/a6596447-1335-4ce0-8d05-4f8ff15ea382-kube-api-access-xpdbh\") pod \"multus-crcwz\" (UID: \"a6596447-1335-4ce0-8d05-4f8ff15ea382\") " pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.251165 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.251145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzlqv\" (UniqueName: \"kubernetes.io/projected/1c6f1795-9275-4b38-ba50-f2653f2b58dc-kube-api-access-gzlqv\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:42.346134 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.346054    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:25:42.354306 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.354281    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f51959_8dc3_4814_a455_b557cfedba90.slice/crio-e632ad90b0286366365d69b21e54582d8c70653e0410e1832d47bff3fbd32621 WatchSource:0}: Error finding container e632ad90b0286366365d69b21e54582d8c70653e0410e1832d47bff3fbd32621: Status 404 returned error can't find the container with id e632ad90b0286366365d69b21e54582d8c70653e0410e1832d47bff3fbd32621
Jul 28 01:25:42.371304 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.371285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d"
Jul 28 01:25:42.378253 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.378233    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7f2e29_b4be_483b_a01e_ec496936f036.slice/crio-9b33ba5400581b375c9b47ad48a6274f21687c77231a986299ad0d8eff5f36d5 WatchSource:0}: Error finding container 9b33ba5400581b375c9b47ad48a6274f21687c77231a986299ad0d8eff5f36d5: Status 404 returned error can't find the container with id 9b33ba5400581b375c9b47ad48a6274f21687c77231a986299ad0d8eff5f36d5
Jul 28 01:25:42.379297 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.379282    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xp8sj"
Jul 28 01:25:42.386498 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.386458    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8c170e_1f5f_450c_8784_db3bef797f61.slice/crio-3b3e9c5b7d21f3251ccde8638673c856d9263a8471694d23b00d83fdc7ac884f WatchSource:0}: Error finding container 3b3e9c5b7d21f3251ccde8638673c856d9263a8471694d23b00d83fdc7ac884f: Status 404 returned error can't find the container with id 3b3e9c5b7d21f3251ccde8638673c856d9263a8471694d23b00d83fdc7ac884f
Jul 28 01:25:42.410272 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.410239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kkv68"
Jul 28 01:25:42.416213 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.416192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w6knn"
Jul 28 01:25:42.417120 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.417098    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffaa729e_9e90_4a5d_825c_d61951b1ca92.slice/crio-59ab774115ecdcaec9a4ac82cc7b238a33d1d4d240181b55e33ebe802b84d681 WatchSource:0}: Error finding container 59ab774115ecdcaec9a4ac82cc7b238a33d1d4d240181b55e33ebe802b84d681: Status 404 returned error can't find the container with id 59ab774115ecdcaec9a4ac82cc7b238a33d1d4d240181b55e33ebe802b84d681
Jul 28 01:25:42.423939 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.423919    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb004963_7a39_48ea_be4f_286fada36b32.slice/crio-3a4850ee0ba3e68ca6aa8d2fd953fa6dd3450e64039d4cfddd69658c35ab9021 WatchSource:0}: Error finding container 3a4850ee0ba3e68ca6aa8d2fd953fa6dd3450e64039d4cfddd69658c35ab9021: Status 404 returned error can't find the container with id 3a4850ee0ba3e68ca6aa8d2fd953fa6dd3450e64039d4cfddd69658c35ab9021
Jul 28 01:25:42.425712 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.425697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-crcwz"
Jul 28 01:25:42.432315 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.432298    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6596447_1335_4ce0_8d05_4f8ff15ea382.slice/crio-f806a31eae2e31ad3320dc5f66019dbed85e52b832a12ddde7fd032e3dff39f0 WatchSource:0}: Error finding container f806a31eae2e31ad3320dc5f66019dbed85e52b832a12ddde7fd032e3dff39f0: Status 404 returned error can't find the container with id f806a31eae2e31ad3320dc5f66019dbed85e52b832a12ddde7fd032e3dff39f0
Jul 28 01:25:42.433163 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.433142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5mjjb"
Jul 28 01:25:42.435182 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.435164    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:42.440250 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.440232    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65669d60_5522_49e7_a490_1c2ac00830b1.slice/crio-4bab4266c0c680e2e592e6f235627c4a7bff4fea13c9a033060529f5d8b06ff5 WatchSource:0}: Error finding container 4bab4266c0c680e2e592e6f235627c4a7bff4fea13c9a033060529f5d8b06ff5: Status 404 returned error can't find the container with id 4bab4266c0c680e2e592e6f235627c4a7bff4fea13c9a033060529f5d8b06ff5
Jul 28 01:25:42.455866 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.455846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4vkm"
Jul 28 01:25:42.461091 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.461077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:25:42.463690 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.463672    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69084ae6_a530_4718_a6bc_2d9481b7c76c.slice/crio-820ef49b7fa9e834e677034c55de2868d659221429330e73aeb5c0f9c55a609f WatchSource:0}: Error finding container 820ef49b7fa9e834e677034c55de2868d659221429330e73aeb5c0f9c55a609f: Status 404 returned error can't find the container with id 820ef49b7fa9e834e677034c55de2868d659221429330e73aeb5c0f9c55a609f
Jul 28 01:25:42.467975 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:25:42.467956    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ad062e_f0ff_45cc_8a4a_db2e2a833a17.slice/crio-97f08f4805afa9791fed3b340ecde7ccb7616fd319d27af29d38d7f5eb80a702 WatchSource:0}: Error finding container 97f08f4805afa9791fed3b340ecde7ccb7616fd319d27af29d38d7f5eb80a702: Status 404 returned error can't find the container with id 97f08f4805afa9791fed3b340ecde7ccb7616fd319d27af29d38d7f5eb80a702
Jul 28 01:25:42.748275 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.747201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:42.748275 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.747266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:42.748275 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.747390    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:42.748275 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.747450    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs podName:1c6f1795-9275-4b38-ba50-f2653f2b58dc nodeName:}" failed. No retries permitted until 2026-07-28 01:25:43.747431245 +0000 UTC m=+3.053920028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs") pod "network-metrics-daemon-xskmq" (UID: "1c6f1795-9275-4b38-ba50-f2653f2b58dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:42.748275 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.747860    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:42.748275 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.747911    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret podName:fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:43.747896393 +0000 UTC m=+3.054385159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret") pod "global-pull-secret-syncer-gd4x8" (UID: "fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:42.775146 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.775114    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:42.848224 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:42.848191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:42.848379 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.848344    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:42.848379 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.848365    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:42.848379 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.848376    2579 projected.go:196] Error preparing data for projected volume kube-api-access-t8s2h for pod openshift-network-diagnostics/network-check-target-2x9gm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:42.848556 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:42.848453    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h podName:f88f1a31-21c3-45c5-8cc3-f44b91806734 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:43.848421579 +0000 UTC m=+3.154910341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8s2h" (UniqueName: "kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h") pod "network-check-target-2x9gm" (UID: "f88f1a31-21c3-45c5-8cc3-f44b91806734") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:43.076805 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.076703    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:42 +0000 UTC" deadline="2028-01-20 23:01:02.83659283 +0000 UTC"
Jul 28 01:25:43.076805 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.076762    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13005h35m19.759834102s"
Jul 28 01:25:43.193156 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.193044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-crcwz" event={"ID":"a6596447-1335-4ce0-8d05-4f8ff15ea382","Type":"ContainerStarted","Data":"f806a31eae2e31ad3320dc5f66019dbed85e52b832a12ddde7fd032e3dff39f0"}
Jul 28 01:25:43.209629 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.209587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kkv68" event={"ID":"ffaa729e-9e90-4a5d-825c-d61951b1ca92","Type":"ContainerStarted","Data":"59ab774115ecdcaec9a4ac82cc7b238a33d1d4d240181b55e33ebe802b84d681"}
Jul 28 01:25:43.227739 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.227700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal" event={"ID":"d1c22b9cc929456a299125322077be27","Type":"ContainerStarted","Data":"8d4d8f0210b380b8af7e3011250758172013a201d4eb04b4c2a9f216c46249db"}
Jul 28 01:25:43.231500 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.231456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal" event={"ID":"28b62562efbbd88ae10aada4b859f4ab","Type":"ContainerStarted","Data":"34cb474a55fd59d7b2d247b21fa8d548bfa4a879d1d0a6a79db14886bdc22d15"}
Jul 28 01:25:43.236399 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.236348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6knn" event={"ID":"db004963-7a39-48ea-be4f-286fada36b32","Type":"ContainerStarted","Data":"3a4850ee0ba3e68ca6aa8d2fd953fa6dd3450e64039d4cfddd69658c35ab9021"}
Jul 28 01:25:43.261900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.261863    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xp8sj" event={"ID":"7a8c170e-1f5f-450c-8784-db3bef797f61","Type":"ContainerStarted","Data":"3b3e9c5b7d21f3251ccde8638673c856d9263a8471694d23b00d83fdc7ac884f"}
Jul 28 01:25:43.271636 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.271603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d" event={"ID":"ad7f2e29-b4be-483b-a01e-ec496936f036","Type":"ContainerStarted","Data":"9b33ba5400581b375c9b47ad48a6274f21687c77231a986299ad0d8eff5f36d5"}
Jul 28 01:25:43.277904 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.277869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jn5d4" event={"ID":"63f51959-8dc3-4814-a455-b557cfedba90","Type":"ContainerStarted","Data":"e632ad90b0286366365d69b21e54582d8c70653e0410e1832d47bff3fbd32621"}
Jul 28 01:25:43.300728 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.300689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"97f08f4805afa9791fed3b340ecde7ccb7616fd319d27af29d38d7f5eb80a702"}
Jul 28 01:25:43.314975 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.314942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4vkm" event={"ID":"69084ae6-a530-4718-a6bc-2d9481b7c76c","Type":"ContainerStarted","Data":"820ef49b7fa9e834e677034c55de2868d659221429330e73aeb5c0f9c55a609f"}
Jul 28 01:25:43.322822 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.322524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5mjjb" event={"ID":"65669d60-5522-49e7-a490-1c2ac00830b1","Type":"ContainerStarted","Data":"4bab4266c0c680e2e592e6f235627c4a7bff4fea13c9a033060529f5d8b06ff5"}
Jul 28 01:25:43.757474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.757439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:43.757667 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.757535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:43.757667 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:43.757663    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:43.757785 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:43.757723    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret podName:fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:45.757704077 +0000 UTC m=+5.064192837 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret") pod "global-pull-secret-syncer-gd4x8" (UID: "fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:43.758176 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:43.758157    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:43.758256 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:43.758213    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs podName:1c6f1795-9275-4b38-ba50-f2653f2b58dc nodeName:}" failed. No retries permitted until 2026-07-28 01:25:45.758196742 +0000 UTC m=+5.064685504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs") pod "network-metrics-daemon-xskmq" (UID: "1c6f1795-9275-4b38-ba50-f2653f2b58dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:43.858243 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:43.858203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:43.858422 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:43.858413    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:43.858527 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:43.858434    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:43.858527 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:43.858446    2579 projected.go:196] Error preparing data for projected volume kube-api-access-t8s2h for pod openshift-network-diagnostics/network-check-target-2x9gm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:43.858527 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:43.858517    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h podName:f88f1a31-21c3-45c5-8cc3-f44b91806734 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:45.858499275 +0000 UTC m=+5.164988037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8s2h" (UniqueName: "kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h") pod "network-check-target-2x9gm" (UID: "f88f1a31-21c3-45c5-8cc3-f44b91806734") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:44.077891 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:44.077795    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:42 +0000 UTC" deadline="2028-03-30 19:06:18.319401747 +0000 UTC"
Jul 28 01:25:44.077891 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:44.077840    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14681h40m34.241566565s"
Jul 28 01:25:44.158948 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:44.158917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:44.159260 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:44.159050    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:44.159589 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:44.159554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:44.159683 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:44.159661    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:25:44.159786 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:44.159757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:44.159868 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:44.159849    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:44.750552 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:44.750521    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:45.066587 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:45.066512    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:45.775679 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:45.775594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:45.775679 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:45.775669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:45.776258 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:45.775777    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:45.776258 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:45.775836    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret podName:fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:49.775822832 +0000 UTC m=+9.082311591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret") pod "global-pull-secret-syncer-gd4x8" (UID: "fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:45.776258 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:45.775849    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:45.776258 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:45.775908    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs podName:1c6f1795-9275-4b38-ba50-f2653f2b58dc nodeName:}" failed. No retries permitted until 2026-07-28 01:25:49.775891669 +0000 UTC m=+9.082380439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs") pod "network-metrics-daemon-xskmq" (UID: "1c6f1795-9275-4b38-ba50-f2653f2b58dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:45.876314 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:45.876278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:45.876500 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:45.876462    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:45.876500 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:45.876493    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:45.876614 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:45.876506    2579 projected.go:196] Error preparing data for projected volume kube-api-access-t8s2h for pod openshift-network-diagnostics/network-check-target-2x9gm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:45.876614 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:45.876572    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h podName:f88f1a31-21c3-45c5-8cc3-f44b91806734 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:49.876552161 +0000 UTC m=+9.183040937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8s2h" (UniqueName: "kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h") pod "network-check-target-2x9gm" (UID: "f88f1a31-21c3-45c5-8cc3-f44b91806734") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:46.158057 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:46.157988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:46.158216 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:46.158106    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:25:46.158216 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:46.158136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:46.158322 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:46.158240    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:46.158367 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:46.158323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:46.158421 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:46.158403    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:48.158878 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:48.158841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:48.158878 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:48.158876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:48.159427 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:48.158944    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:25:48.159427 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:48.159014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:48.159427 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:48.159108    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:48.159427 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:48.159212    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:49.812558 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:49.812242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:49.812558 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:49.812328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:49.812558 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:49.812466    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:49.812558 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:49.812544    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret podName:fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:57.812525258 +0000 UTC m=+17.119014040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret") pod "global-pull-secret-syncer-gd4x8" (UID: "fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:49.813120 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:49.812628    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:49.813120 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:49.812696    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs podName:1c6f1795-9275-4b38-ba50-f2653f2b58dc nodeName:}" failed. No retries permitted until 2026-07-28 01:25:57.812676689 +0000 UTC m=+17.119165464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs") pod "network-metrics-daemon-xskmq" (UID: "1c6f1795-9275-4b38-ba50-f2653f2b58dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:49.913580 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:49.913542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:49.913764 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:49.913711    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:49.913764 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:49.913729    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:49.913764 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:49.913742    2579 projected.go:196] Error preparing data for projected volume kube-api-access-t8s2h for pod openshift-network-diagnostics/network-check-target-2x9gm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:49.913921 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:49.913804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h podName:f88f1a31-21c3-45c5-8cc3-f44b91806734 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:57.913785802 +0000 UTC m=+17.220274580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8s2h" (UniqueName: "kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h") pod "network-check-target-2x9gm" (UID: "f88f1a31-21c3-45c5-8cc3-f44b91806734") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:50.158630 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:50.158593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:50.158828 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:50.158736    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:50.158828 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:50.158754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:50.158950 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:50.158849    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:25:50.158950 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:50.158874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:50.158950 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:50.158935    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:52.158527 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:52.158474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:52.158985 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:52.158529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:52.158985 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:52.158529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:52.158985 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:52.158627    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:52.158985 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:52.158715    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:52.158985 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:52.158804    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:25:54.158393 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:54.158355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:54.158837 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:54.158527    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:54.158837 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:54.158537    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:25:54.158837 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:54.158652    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:54.158837 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:54.158705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:54.158837 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:54.158757    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:56.158417 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:56.158380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:56.158417 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:56.158408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:56.158417 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:56.158421    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:56.158946 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:56.158527    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:56.158946 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:56.158663    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:25:56.158946 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:56.158768    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:57.871639 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:57.871611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:57.872128 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:57.871666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:57.872128 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:57.871789    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:57.872128 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:57.871866    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret podName:fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:13.871846553 +0000 UTC m=+33.178335334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret") pod "global-pull-secret-syncer-gd4x8" (UID: "fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:57.872128 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:57.871798    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:57.872128 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:57.871936    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs podName:1c6f1795-9275-4b38-ba50-f2653f2b58dc nodeName:}" failed. No retries permitted until 2026-07-28 01:26:13.871919084 +0000 UTC m=+33.178407844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs") pod "network-metrics-daemon-xskmq" (UID: "1c6f1795-9275-4b38-ba50-f2653f2b58dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:57.972286 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:57.972199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:57.972448 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:57.972343    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:57.972448 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:57.972362    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:57.972448 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:57.972375    2579 projected.go:196] Error preparing data for projected volume kube-api-access-t8s2h for pod openshift-network-diagnostics/network-check-target-2x9gm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:57.972448 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:57.972444    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h podName:f88f1a31-21c3-45c5-8cc3-f44b91806734 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:13.972425636 +0000 UTC m=+33.278914418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8s2h" (UniqueName: "kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h") pod "network-check-target-2x9gm" (UID: "f88f1a31-21c3-45c5-8cc3-f44b91806734") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:58.158273 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:58.158240    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:25:58.158452 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:58.158286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:25:58.158452 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:25:58.158256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:25:58.158452 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:58.158374    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:25:58.158657 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:58.158453    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:25:58.158657 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:25:58.158549    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:00.158437 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:00.158390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:00.158893 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:00.158401    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:00.158893 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:00.158545    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:00.158893 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:00.158410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:00.158893 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:00.158624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:00.158893 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:00.158698    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:02.158864 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.158826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:02.159363 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.158907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:02.159363 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.159000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:02.159363 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:02.159019    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:02.159363 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:02.159077    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:02.159363 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:02.159147    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:02.360228 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.360197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"065b5ff200273bc0a9ab0923fe7ffad7cfcf65d686d2763d23641f8946299155"}
Jul 28 01:26:02.362092 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.362066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-crcwz" event={"ID":"a6596447-1335-4ce0-8d05-4f8ff15ea382","Type":"ContainerStarted","Data":"b277e4c09052efefe75b4ce28c5d69560a604cc3eb723d922454114b862e809a"}
Jul 28 01:26:02.363457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.363426    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal" event={"ID":"d1c22b9cc929456a299125322077be27","Type":"ContainerStarted","Data":"62c0c086be6243321814d9504c90fd7708c74249ce368de02eac53fb9c9b4d0b"}
Jul 28 01:26:02.364921 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.364899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xp8sj" event={"ID":"7a8c170e-1f5f-450c-8784-db3bef797f61","Type":"ContainerStarted","Data":"ea6496732d2175b115664ff032307e8511ced19c885c4a09219f04a14ddc6727"}
Jul 28 01:26:02.398131 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.397983    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-crcwz" podStartSLOduration=2.121321352 podStartE2EDuration="21.39796231s" totalImagesPullingTime="19.276640958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:02.397050308 +0000 UTC m=+21.703539091" watchObservedRunningTime="2026-07-28 01:26:02.39796231 +0000 UTC m=+21.704451104"
Jul 28 01:26:02.417517 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.417451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-30.ec2.internal" podStartSLOduration=21.417435742 podStartE2EDuration="21.417435742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:02.41702247 +0000 UTC m=+21.723511267" watchObservedRunningTime="2026-07-28 01:26:02.417435742 +0000 UTC m=+21.723924522"
Jul 28 01:26:02.442261 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:02.441873    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xp8sj" podStartSLOduration=2.148973929 podStartE2EDuration="21.441852286s" totalImagesPullingTime="19.292878357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:02.441233882 +0000 UTC m=+21.747722696" watchObservedRunningTime="2026-07-28 01:26:02.441852286 +0000 UTC m=+21.748341069"
Jul 28 01:26:03.368584 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.368367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d" event={"ID":"ad7f2e29-b4be-483b-a01e-ec496936f036","Type":"ContainerStarted","Data":"a6c6ec0287eb43fc00ceb63f779f1eafb2e3fe8fa1e60bfba4a4760aa83d48d9"}
Jul 28 01:26:03.369651 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.369627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jn5d4" event={"ID":"63f51959-8dc3-4814-a455-b557cfedba90","Type":"ContainerStarted","Data":"ad31ba69fd71b0f92c6f431d6479b11ff8f03b8820de6ad52b52a6ba7fb77e91"}
Jul 28 01:26:03.372166 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.372147    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:26:03.372446 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.372427    2579 generic.go:356] "Generic (PLEG): container finished" podID="d0ad062e-f0ff-45cc-8a4a-db2e2a833a17" containerID="c33ce9b8fbf8baed0cbce5a132e5337276013a1e036f74ad87ad982a46230c5b" exitCode=1
Jul 28 01:26:03.372526 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.372506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"6c24ec80df138748bbcc76d819fc80629b5fa0bb58e886f23344f3bfe2f75e43"}
Jul 28 01:26:03.372595 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.372581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"a0198a2331679767fb8a3f69ceea8a77dcd8716351c4cf357dd59f0aaafbb17b"}
Jul 28 01:26:03.372649 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.372636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"66ce02b0ff49b97ea4d4035791ccd0d29edc5ddda36c68469c1f06f5782e149b"}
Jul 28 01:26:03.372704 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.372684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"b50c5183cf389d0fcff185b0100011d7082fd04279963f300a96c8bd793f223e"}
Jul 28 01:26:03.372763 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.372724    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerDied","Data":"c33ce9b8fbf8baed0cbce5a132e5337276013a1e036f74ad87ad982a46230c5b"}
Jul 28 01:26:03.373837 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.373817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4vkm" event={"ID":"69084ae6-a530-4718-a6bc-2d9481b7c76c","Type":"ContainerStarted","Data":"03b9469d8501394989d3cb846e513c340bfd961cbc4c10f256d4c3fee2768674"}
Jul 28 01:26:03.375215 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.375192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5mjjb" event={"ID":"65669d60-5522-49e7-a490-1c2ac00830b1","Type":"ContainerStarted","Data":"5040c811b7cdcca19c01d7d1b1a560672273561492e3bf50f1e3a4edbfeb78a7"}
Jul 28 01:26:03.376527 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.376506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kkv68" event={"ID":"ffaa729e-9e90-4a5d-825c-d61951b1ca92","Type":"ContainerStarted","Data":"5e509d745afdc131d3866ec58256665e77fbfe182a59c95ca7a998b8dd125641"}
Jul 28 01:26:03.377845 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.377826    2579 generic.go:356] "Generic (PLEG): container finished" podID="28b62562efbbd88ae10aada4b859f4ab" containerID="5c296b4b48224474a036ef6e7bb73734107514f1377e55f933805d892ff5d9bd" exitCode=0
Jul 28 01:26:03.377925 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.377887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal" event={"ID":"28b62562efbbd88ae10aada4b859f4ab","Type":"ContainerDied","Data":"5c296b4b48224474a036ef6e7bb73734107514f1377e55f933805d892ff5d9bd"}
Jul 28 01:26:03.379866 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.379843    2579 generic.go:356] "Generic (PLEG): container finished" podID="db004963-7a39-48ea-be4f-286fada36b32" containerID="5546531f3322e2674eaced089be1dd158f82daa8525666cd9b0ca77e908df88b" exitCode=0
Jul 28 01:26:03.379955 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.379927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6knn" event={"ID":"db004963-7a39-48ea-be4f-286fada36b32","Type":"ContainerDied","Data":"5546531f3322e2674eaced089be1dd158f82daa8525666cd9b0ca77e908df88b"}
Jul 28 01:26:03.386188 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.386152    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jn5d4" podStartSLOduration=3.08777373 podStartE2EDuration="22.386141221s" totalImagesPullingTime="19.298367491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:03.385825175 +0000 UTC m=+22.692313975" watchObservedRunningTime="2026-07-28 01:26:03.386141221 +0000 UTC m=+22.692630003"
Jul 28 01:26:03.405961 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.405924    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5mjjb" podStartSLOduration=3.166972837 podStartE2EDuration="22.405910279s" totalImagesPullingTime="19.238937442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:03.405687036 +0000 UTC m=+22.712175817" watchObservedRunningTime="2026-07-28 01:26:03.405910279 +0000 UTC m=+22.712399059"
Jul 28 01:26:03.437438 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.437385    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kkv68" podStartSLOduration=3.201777231 podStartE2EDuration="22.437367686s" totalImagesPullingTime="19.235590455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:03.437183999 +0000 UTC m=+22.743672798" watchObservedRunningTime="2026-07-28 01:26:03.437367686 +0000 UTC m=+22.743856467"
Jul 28 01:26:03.451285 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.451244    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g4vkm" podStartSLOduration=3.262056705 podStartE2EDuration="22.451229451s" totalImagesPullingTime="19.189172746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:03.451053299 +0000 UTC m=+22.757542091" watchObservedRunningTime="2026-07-28 01:26:03.451229451 +0000 UTC m=+22.757718232"
Jul 28 01:26:03.599606 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:03.599577    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 01:26:04.102781 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.102644    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T01:26:03.599602278Z","UUID":"a8a7056b-3211-4ccc-8b5e-6f98cc63e0a5","Handler":null,"Name":"","Endpoint":""}
Jul 28 01:26:04.105103 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.105080    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 01:26:04.105238 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.105120    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 01:26:04.158517 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.158468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:04.158672 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.158584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:04.158672 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:04.158589    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:04.158672 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.158589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:04.158770 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:04.158667    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:04.158770 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:04.158735    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:04.385013 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.384972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal" event={"ID":"28b62562efbbd88ae10aada4b859f4ab","Type":"ContainerStarted","Data":"5cdf538c347ed67151e5fffff05f65a4c64a96244b5a76dfbb0c8dc0d45fa9d4"}
Jul 28 01:26:04.388263 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.388230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d" event={"ID":"ad7f2e29-b4be-483b-a01e-ec496936f036","Type":"ContainerStarted","Data":"02573f9cd99f8c77d56030b0b4617f20f6a34eeac1399de20aabc35ce90e810d"}
Jul 28 01:26:04.406452 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:04.406391    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-30.ec2.internal" podStartSLOduration=23.406370524 podStartE2EDuration="23.406370524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:04.405962023 +0000 UTC m=+23.712450805" watchObservedRunningTime="2026-07-28 01:26:04.406370524 +0000 UTC m=+23.712859307"
Jul 28 01:26:05.392532 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:05.392499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d" event={"ID":"ad7f2e29-b4be-483b-a01e-ec496936f036","Type":"ContainerStarted","Data":"d5059c7058f263dbb74dc2d93ff07c38e91a1fdbc8da0cdde1310a47fa02f57e"}
Jul 28 01:26:05.396034 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:05.396007    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:26:05.396425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:05.396402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"dcd936de010762934eec767ffcc1fe67d1d9ce9ec423c8d7f667aecd8782d11a"}
Jul 28 01:26:05.409613 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:05.409568    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvw9d" podStartSLOduration=2.770030846 podStartE2EDuration="24.40955303s" totalImagesPullingTime="21.639522184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:05.409297164 +0000 UTC m=+24.715785956" watchObservedRunningTime="2026-07-28 01:26:05.40955303 +0000 UTC m=+24.716041810"
Jul 28 01:26:06.158528 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:06.158477    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:06.158528 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:06.158520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:06.158768 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:06.158518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:06.158768 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:06.158600    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:06.158768 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:06.158692    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:06.158929 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:06.158768    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:07.153207 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:07.153183    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:26:07.153896 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:07.153878    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:26:07.400077 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:07.400006    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:26:07.400701 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:07.400680    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jn5d4"
Jul 28 01:26:08.158283 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.158079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:08.158938 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.158131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:08.158938 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:08.158365    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:08.158938 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.158132    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:08.158938 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:08.158420    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:08.158938 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:08.158526    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:08.403715 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.403679    2579 generic.go:356] "Generic (PLEG): container finished" podID="db004963-7a39-48ea-be4f-286fada36b32" containerID="b0af4c55db4f3e4b507cd3bd74b43672f504f2c2f19406cce2df0986915a333a" exitCode=0
Jul 28 01:26:08.403895 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.403764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6knn" event={"ID":"db004963-7a39-48ea-be4f-286fada36b32","Type":"ContainerDied","Data":"b0af4c55db4f3e4b507cd3bd74b43672f504f2c2f19406cce2df0986915a333a"}
Jul 28 01:26:08.407426 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.407409    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:26:08.407742 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.407719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"a33f8cddd4ea2fa2b9072eb6441f723c653ee64c1739a4fb952c3d40fa753890"}
Jul 28 01:26:08.407981 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.407960    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:26:08.408079 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.408032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:26:08.408129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.408081    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:26:08.408166 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.408154    2579 scope.go:117] "RemoveContainer" containerID="c33ce9b8fbf8baed0cbce5a132e5337276013a1e036f74ad87ad982a46230c5b"
Jul 28 01:26:08.423294 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.423276    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:26:08.423383 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:08.423375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:26:09.413270 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.413250    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:26:09.413803 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.413774    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" event={"ID":"d0ad062e-f0ff-45cc-8a4a-db2e2a833a17","Type":"ContainerStarted","Data":"2c7e85bae3c5494ab3afdee3428895ffb0ed95e6cb514d0b185025597c94fae8"}
Jul 28 01:26:09.445250 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.445204    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx" podStartSLOduration=8.671820853 podStartE2EDuration="28.445190043s" totalImagesPullingTime="19.77336919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:09.44472031 +0000 UTC m=+28.751209091" watchObservedRunningTime="2026-07-28 01:26:09.445190043 +0000 UTC m=+28.751678824"
Jul 28 01:26:09.873047 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.872858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gd4x8"]
Jul 28 01:26:09.873214 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.873160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:09.873265 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:09.873253    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:09.879172 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.879132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xskmq"]
Jul 28 01:26:09.879293 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.879261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:09.879390 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:09.879369    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:09.879769 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.879745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2x9gm"]
Jul 28 01:26:09.879887 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:09.879824    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:09.879948 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:09.879905    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:10.075066 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:10.074997    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mjjb_65669d60-5522-49e7-a490-1c2ac00830b1/dns-node-resolver/0.log"
Jul 28 01:26:10.418297 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:10.418267    2579 generic.go:356] "Generic (PLEG): container finished" podID="db004963-7a39-48ea-be4f-286fada36b32" containerID="f40fd34a65291e2e5dd72b3044e0fc7ad737246aaf898b334f2edd8ed234e992" exitCode=0
Jul 28 01:26:10.418792 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:10.418352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6knn" event={"ID":"db004963-7a39-48ea-be4f-286fada36b32","Type":"ContainerDied","Data":"f40fd34a65291e2e5dd72b3044e0fc7ad737246aaf898b334f2edd8ed234e992"}
Jul 28 01:26:11.159259 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:11.159234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:11.159409 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:11.159316    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:11.159409 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:11.159322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:11.159409 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:11.159384    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:11.249255 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:11.249233    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kkv68_ffaa729e-9e90-4a5d-825c-d61951b1ca92/node-ca/0.log"
Jul 28 01:26:12.158110 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:12.158031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:12.158635 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:12.158136    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:12.425975 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:12.425892    2579 generic.go:356] "Generic (PLEG): container finished" podID="db004963-7a39-48ea-be4f-286fada36b32" containerID="406ccb36405b67977cefdca51ef54860f5c18bd83848115bbfcbe49c181bc07f" exitCode=0
Jul 28 01:26:12.425975 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:12.425948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6knn" event={"ID":"db004963-7a39-48ea-be4f-286fada36b32","Type":"ContainerDied","Data":"406ccb36405b67977cefdca51ef54860f5c18bd83848115bbfcbe49c181bc07f"}
Jul 28 01:26:13.158699 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:13.158662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:13.159281 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.158786    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:13.159281 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:13.158843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:13.159281 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.158949    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:13.891805 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:13.891767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:13.891983 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:13.891858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:13.891983 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.891940    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:13.891983 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.891973    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:13.892101 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.892014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs podName:1c6f1795-9275-4b38-ba50-f2653f2b58dc nodeName:}" failed. No retries permitted until 2026-07-28 01:26:45.891990952 +0000 UTC m=+65.198479716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs") pod "network-metrics-daemon-xskmq" (UID: "1c6f1795-9275-4b38-ba50-f2653f2b58dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:13.892101 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.892034    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret podName:fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:45.892024116 +0000 UTC m=+65.198512903 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret") pod "global-pull-secret-syncer-gd4x8" (UID: "fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:13.992466 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:13.992426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:13.992643 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.992600    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:13.992643 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.992622    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:13.992643 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.992635    2579 projected.go:196] Error preparing data for projected volume kube-api-access-t8s2h for pod openshift-network-diagnostics/network-check-target-2x9gm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:13.992743 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:13.992687    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h podName:f88f1a31-21c3-45c5-8cc3-f44b91806734 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:45.9926723 +0000 UTC m=+65.299161064 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t8s2h" (UniqueName: "kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h") pod "network-check-target-2x9gm" (UID: "f88f1a31-21c3-45c5-8cc3-f44b91806734") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:14.158310 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:14.158230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:14.158464 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:14.158363    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:15.158928 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:15.158895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:15.159546 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:15.158895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:15.159546 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:15.159032    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:15.159546 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:15.159109    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:16.158759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:16.158729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:16.158911 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:16.158862    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:17.158067 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:17.158040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:17.158550 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:17.158149    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:17.158550 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:17.158233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:17.158550 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:17.158333    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:17.439893 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:17.439805    2579 generic.go:356] "Generic (PLEG): container finished" podID="db004963-7a39-48ea-be4f-286fada36b32" containerID="dab9c29634d03a5cb469d974752a6ac5c8e514b8fc05bbfb14ffab30f94e8f51" exitCode=0
Jul 28 01:26:17.439893 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:17.439864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6knn" event={"ID":"db004963-7a39-48ea-be4f-286fada36b32","Type":"ContainerDied","Data":"dab9c29634d03a5cb469d974752a6ac5c8e514b8fc05bbfb14ffab30f94e8f51"}
Jul 28 01:26:18.158269 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:18.158236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:18.158629 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:18.158342    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:18.445113 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:18.445040    2579 generic.go:356] "Generic (PLEG): container finished" podID="db004963-7a39-48ea-be4f-286fada36b32" containerID="b8e2fa48ec8dd7acea6b80f9f4f981f5497069831ddc0755c69d0316ad42f470" exitCode=0
Jul 28 01:26:18.445249 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:18.445098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6knn" event={"ID":"db004963-7a39-48ea-be4f-286fada36b32","Type":"ContainerDied","Data":"b8e2fa48ec8dd7acea6b80f9f4f981f5497069831ddc0755c69d0316ad42f470"}
Jul 28 01:26:19.158257 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:19.158082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:19.158475 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:19.158118    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:19.158475 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:19.158327    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:19.158475 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:19.158414    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:19.450631 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:19.450546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w6knn" event={"ID":"db004963-7a39-48ea-be4f-286fada36b32","Type":"ContainerStarted","Data":"c94a519d2fa05fc5bf4fcf608c8b8e773d9603d8c8fbd5e86b0ceddbaf2a1e70"}
Jul 28 01:26:19.475776 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:19.475737    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w6knn" podStartSLOduration=8.819198233 podStartE2EDuration="38.475721606s" totalImagesPullingTime="29.656523373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:19.475201237 +0000 UTC m=+38.781690017" watchObservedRunningTime="2026-07-28 01:26:19.475721606 +0000 UTC m=+38.782210387"
Jul 28 01:26:20.158759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:20.158729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:20.159106 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:20.158831    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:21.159197 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:21.159168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:21.159614 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:21.159244    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:21.159614 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:21.159352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:21.159614 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:21.159463    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:22.158646 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:22.158612    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:22.158825 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:22.158720    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:23.158662 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:23.158626    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:23.159003 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:23.158730    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:23.159003 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:23.158782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:23.159003 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:23.158863    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:24.158538 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:24.158508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:24.158711 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:24.158608    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:25.158188 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:25.158152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:25.158456 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:25.158287    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:25.158456 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:25.158358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:25.158456 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:25.158449    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:26.158185 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:26.158153    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:26.158609 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:26.158258    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:27.158305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:27.158265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:27.158801 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:27.158350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:27.158801 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:27.158366    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:27.158801 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:27.158510    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:28.158658 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:28.158632    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:28.159024 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:28.158731    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:29.157890 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:29.157862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:29.157890 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:29.157878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:29.158113 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:29.157963    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:29.158113 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:29.158065    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:30.157993 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:30.157965    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:30.158370 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:30.158062    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:31.159133 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:31.159100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:31.159612 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:31.159201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:31.159612 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:31.159236    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:31.159612 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:31.159271    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:32.158309 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:32.158271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:32.158517 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:32.158388    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xskmq" podUID="1c6f1795-9275-4b38-ba50-f2653f2b58dc"
Jul 28 01:26:33.158462 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:33.158428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:33.158849 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:33.158429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:33.158849 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:33.158548    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2x9gm" podUID="f88f1a31-21c3-45c5-8cc3-f44b91806734"
Jul 28 01:26:33.158849 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:33.158603    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gd4x8" podUID="fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021"
Jul 28 01:26:34.086647 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.086620    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-30.ec2.internal" event="NodeReady"
Jul 28 01:26:34.086808 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.086725    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 01:26:34.125861 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.125831    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c6d86cc9-69rg5"]
Jul 28 01:26:34.127650 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.127635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.130500 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.130463    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 01:26:34.130644 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.130545    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gmm4s\""
Jul 28 01:26:34.131271 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.131238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 01:26:34.131369 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.131287    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 01:26:34.137582 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.137565    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 01:26:34.143837 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.143817    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m789s"]
Jul 28 01:26:34.148164 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.148137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c6d86cc9-69rg5"]
Jul 28 01:26:34.148266 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.148236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.148434 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.148413    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4wn9f"]
Jul 28 01:26:34.150403 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.150386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4wn9f"
Jul 28 01:26:34.153025 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.152964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 01:26:34.153025 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.152995    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 01:26:34.153340 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.153326    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zgp9f\""
Jul 28 01:26:34.153431 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.153407    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 01:26:34.153715 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.153697    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 01:26:34.153794 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.153752    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 01:26:34.153794 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.153761    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6wwkj\""
Jul 28 01:26:34.153794 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.153789    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 01:26:34.153943 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.153862    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 01:26:34.157713 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4498e9e-3eff-445b-b3af-23d0ad786c73-trusted-ca\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.157812 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-bound-sa-token\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.157812 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4498e9e-3eff-445b-b3af-23d0ad786c73-image-registry-private-configuration\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.157812 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:34.157812 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-registry-tls\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.158013 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4498e9e-3eff-445b-b3af-23d0ad786c73-registry-certificates\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.158013 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4498e9e-3eff-445b-b3af-23d0ad786c73-ca-trust-extracted\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.158013 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4498e9e-3eff-445b-b3af-23d0ad786c73-installation-pull-secrets\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.158013 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.157941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rhzw\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-kube-api-access-9rhzw\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.160531 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.160511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:34.160921 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.160749    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8db89\""
Jul 28 01:26:34.170692 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.170672    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m789s"]
Jul 28 01:26:34.173020 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.173003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4wn9f"]
Jul 28 01:26:34.241379 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.241348    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pbrls"]
Jul 28 01:26:34.243196 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.243183    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.246686 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.246668    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hjm4w\""
Jul 28 01:26:34.246790 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.246708    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 01:26:34.246790 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.246767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 01:26:34.258987 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.258958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rhzw\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-kube-api-access-9rhzw\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.259094 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.258996    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xjv\" (UniqueName: \"kubernetes.io/projected/e2fe9fdd-797b-4c19-91c2-91218d0bf2ca-kube-api-access-b4xjv\") pod \"ingress-canary-4wn9f\" (UID: \"e2fe9fdd-797b-4c19-91c2-91218d0bf2ca\") " pod="openshift-ingress-canary/ingress-canary-4wn9f"
Jul 28 01:26:34.259094 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4498e9e-3eff-445b-b3af-23d0ad786c73-trusted-ca\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.259094 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-crio-socket\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.259094 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-bound-sa-token\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.259296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4498e9e-3eff-445b-b3af-23d0ad786c73-image-registry-private-configuration\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.259296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2fe9fdd-797b-4c19-91c2-91218d0bf2ca-cert\") pod \"ingress-canary-4wn9f\" (UID: \"e2fe9fdd-797b-4c19-91c2-91218d0bf2ca\") " pod="openshift-ingress-canary/ingress-canary-4wn9f"
Jul 28 01:26:34.259296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-data-volume\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.259296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74sz\" (UniqueName: \"kubernetes.io/projected/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-kube-api-access-x74sz\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.259296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.259296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.259608 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-registry-tls\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.259608 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4498e9e-3eff-445b-b3af-23d0ad786c73-registry-certificates\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.259608 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4498e9e-3eff-445b-b3af-23d0ad786c73-ca-trust-extracted\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.259608 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4498e9e-3eff-445b-b3af-23d0ad786c73-installation-pull-secrets\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.259798 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.259732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4498e9e-3eff-445b-b3af-23d0ad786c73-ca-trust-extracted\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.260028 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.260009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4498e9e-3eff-445b-b3af-23d0ad786c73-trusted-ca\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.260088 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.260066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4498e9e-3eff-445b-b3af-23d0ad786c73-registry-certificates\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.262981 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.262958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4498e9e-3eff-445b-b3af-23d0ad786c73-image-registry-private-configuration\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.263094 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.262987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4498e9e-3eff-445b-b3af-23d0ad786c73-installation-pull-secrets\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.263201 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.263181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-registry-tls\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.266803 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.266782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pbrls"]
Jul 28 01:26:34.272975 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.272955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-bound-sa-token\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.273502 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.273464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rhzw\" (UniqueName: \"kubernetes.io/projected/a4498e9e-3eff-445b-b3af-23d0ad786c73-kube-api-access-9rhzw\") pod \"image-registry-c6d86cc9-69rg5\" (UID: \"a4498e9e-3eff-445b-b3af-23d0ad786c73\") " pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.360306 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xjv\" (UniqueName: \"kubernetes.io/projected/e2fe9fdd-797b-4c19-91c2-91218d0bf2ca-kube-api-access-b4xjv\") pod \"ingress-canary-4wn9f\" (UID: \"e2fe9fdd-797b-4c19-91c2-91218d0bf2ca\") " pod="openshift-ingress-canary/ingress-canary-4wn9f"
Jul 28 01:26:34.360306 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-crio-socket\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.360555 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227a00b4-064f-4fef-af22-0a95f22a113c-tmp-dir\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.360555 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/227a00b4-064f-4fef-af22-0a95f22a113c-config-volume\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.360555 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9sdg\" (UniqueName: \"kubernetes.io/projected/227a00b4-064f-4fef-af22-0a95f22a113c-kube-api-access-c9sdg\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.360555 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2fe9fdd-797b-4c19-91c2-91218d0bf2ca-cert\") pod \"ingress-canary-4wn9f\" (UID: \"e2fe9fdd-797b-4c19-91c2-91218d0bf2ca\") " pod="openshift-ingress-canary/ingress-canary-4wn9f"
Jul 28 01:26:34.360750 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-data-volume\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.360750 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74sz\" (UniqueName: \"kubernetes.io/projected/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-kube-api-access-x74sz\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.360750 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-crio-socket\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.360750 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.360750 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.360750 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/227a00b4-064f-4fef-af22-0a95f22a113c-metrics-tls\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.361008 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.360914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-data-volume\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.361135 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.361114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.362834 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.362805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.362922 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.362886    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2fe9fdd-797b-4c19-91c2-91218d0bf2ca-cert\") pod \"ingress-canary-4wn9f\" (UID: \"e2fe9fdd-797b-4c19-91c2-91218d0bf2ca\") " pod="openshift-ingress-canary/ingress-canary-4wn9f"
Jul 28 01:26:34.368022 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.368000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74sz\" (UniqueName: \"kubernetes.io/projected/6098fd8c-fce4-47e1-a5a1-bd9a75d1f937-kube-api-access-x74sz\") pod \"insights-runtime-extractor-m789s\" (UID: \"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937\") " pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.368120 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.368100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xjv\" (UniqueName: \"kubernetes.io/projected/e2fe9fdd-797b-4c19-91c2-91218d0bf2ca-kube-api-access-b4xjv\") pod \"ingress-canary-4wn9f\" (UID: \"e2fe9fdd-797b-4c19-91c2-91218d0bf2ca\") " pod="openshift-ingress-canary/ingress-canary-4wn9f"
Jul 28 01:26:34.436639 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.436572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:34.457329 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.457303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m789s"
Jul 28 01:26:34.461552 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.461530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/227a00b4-064f-4fef-af22-0a95f22a113c-config-volume\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.461642 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.461557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9sdg\" (UniqueName: \"kubernetes.io/projected/227a00b4-064f-4fef-af22-0a95f22a113c-kube-api-access-c9sdg\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.461642 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.461579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/227a00b4-064f-4fef-af22-0a95f22a113c-metrics-tls\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.461847 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.461830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227a00b4-064f-4fef-af22-0a95f22a113c-tmp-dir\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.462173 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.462149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/227a00b4-064f-4fef-af22-0a95f22a113c-tmp-dir\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.462289 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.462235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/227a00b4-064f-4fef-af22-0a95f22a113c-config-volume\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.463124 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.463107    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4wn9f"
Jul 28 01:26:34.464388 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.464347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/227a00b4-064f-4fef-af22-0a95f22a113c-metrics-tls\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.479010 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.478985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9sdg\" (UniqueName: \"kubernetes.io/projected/227a00b4-064f-4fef-af22-0a95f22a113c-kube-api-access-c9sdg\") pod \"dns-default-pbrls\" (UID: \"227a00b4-064f-4fef-af22-0a95f22a113c\") " pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.552186 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.552158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:34.601779 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.601744    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c6d86cc9-69rg5"]
Jul 28 01:26:34.605773 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:34.605744    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4498e9e_3eff_445b_b3af_23d0ad786c73.slice/crio-5af2b6ac3e24001bba02eb5a2a8864bc69cda5703eb62be43adb28f5fe092172 WatchSource:0}: Error finding container 5af2b6ac3e24001bba02eb5a2a8864bc69cda5703eb62be43adb28f5fe092172: Status 404 returned error can't find the container with id 5af2b6ac3e24001bba02eb5a2a8864bc69cda5703eb62be43adb28f5fe092172
Jul 28 01:26:34.683822 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.683793    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pbrls"]
Jul 28 01:26:34.686282 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:34.686260    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227a00b4_064f_4fef_af22_0a95f22a113c.slice/crio-9becae7a5615ac7fba33db0162e092e45563eaae50b0ba123a00224cbd16693e WatchSource:0}: Error finding container 9becae7a5615ac7fba33db0162e092e45563eaae50b0ba123a00224cbd16693e: Status 404 returned error can't find the container with id 9becae7a5615ac7fba33db0162e092e45563eaae50b0ba123a00224cbd16693e
Jul 28 01:26:34.816526 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.816501    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4wn9f"]
Jul 28 01:26:34.819078 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:34.819042    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fe9fdd_797b_4c19_91c2_91218d0bf2ca.slice/crio-d5a4b6d12689d68bb29b0c01d831bda2234022bdc5f1cc9d96e0f591bf6f84f4 WatchSource:0}: Error finding container d5a4b6d12689d68bb29b0c01d831bda2234022bdc5f1cc9d96e0f591bf6f84f4: Status 404 returned error can't find the container with id d5a4b6d12689d68bb29b0c01d831bda2234022bdc5f1cc9d96e0f591bf6f84f4
Jul 28 01:26:34.819799 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:34.819777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m789s"]
Jul 28 01:26:34.825288 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:34.825177    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6098fd8c_fce4_47e1_a5a1_bd9a75d1f937.slice/crio-6bcf74110052c8db67cdb92b9f99f9f9c83c868b23f80bc59a91904306272cfa WatchSource:0}: Error finding container 6bcf74110052c8db67cdb92b9f99f9f9c83c868b23f80bc59a91904306272cfa: Status 404 returned error can't find the container with id 6bcf74110052c8db67cdb92b9f99f9f9c83c868b23f80bc59a91904306272cfa
Jul 28 01:26:35.158110 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.158083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:35.158277 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.158083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:35.160775 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.160589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:35.160775 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.160707    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:35.160775 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.160712    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4lwsn\""
Jul 28 01:26:35.161283 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.161108    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:35.489449 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.489357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4wn9f" event={"ID":"e2fe9fdd-797b-4c19-91c2-91218d0bf2ca","Type":"ContainerStarted","Data":"d5a4b6d12689d68bb29b0c01d831bda2234022bdc5f1cc9d96e0f591bf6f84f4"}
Jul 28 01:26:35.491918 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.491884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m789s" event={"ID":"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937","Type":"ContainerStarted","Data":"6bcf74110052c8db67cdb92b9f99f9f9c83c868b23f80bc59a91904306272cfa"}
Jul 28 01:26:35.493970 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.493941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbrls" event={"ID":"227a00b4-064f-4fef-af22-0a95f22a113c","Type":"ContainerStarted","Data":"9becae7a5615ac7fba33db0162e092e45563eaae50b0ba123a00224cbd16693e"}
Jul 28 01:26:35.497552 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.497527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c6d86cc9-69rg5" event={"ID":"a4498e9e-3eff-445b-b3af-23d0ad786c73","Type":"ContainerStarted","Data":"2d0d8cabe4840b142351d98d3fc264e1fc39c0b9ccf055384c1cab5ce04edd05"}
Jul 28 01:26:35.497661 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.497573    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c6d86cc9-69rg5" event={"ID":"a4498e9e-3eff-445b-b3af-23d0ad786c73","Type":"ContainerStarted","Data":"5af2b6ac3e24001bba02eb5a2a8864bc69cda5703eb62be43adb28f5fe092172"}
Jul 28 01:26:35.498313 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.498276    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:35.519691 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:35.519621    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c6d86cc9-69rg5" podStartSLOduration=2.519600803 podStartE2EDuration="2.519600803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:35.518511527 +0000 UTC m=+54.825000302" watchObservedRunningTime="2026-07-28 01:26:35.519600803 +0000 UTC m=+54.826089582"
Jul 28 01:26:40.402808 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.402773    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"]
Jul 28 01:26:40.410407 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.410374    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"
Jul 28 01:26:40.412714 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.412691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 01:26:40.412842 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.412762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-nc25z\""
Jul 28 01:26:40.417026 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.416997    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"]
Jul 28 01:26:40.430201 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.430181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbrmx"
Jul 28 01:26:40.512362 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.511890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e937ea66-9f7b-4193-af0d-b2a7f6fe4e54-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9hcc4\" (UID: \"e937ea66-9f7b-4193-af0d-b2a7f6fe4e54\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"
Jul 28 01:26:40.612627 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.612603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e937ea66-9f7b-4193-af0d-b2a7f6fe4e54-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9hcc4\" (UID: \"e937ea66-9f7b-4193-af0d-b2a7f6fe4e54\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"
Jul 28 01:26:40.615475 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.615437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e937ea66-9f7b-4193-af0d-b2a7f6fe4e54-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9hcc4\" (UID: \"e937ea66-9f7b-4193-af0d-b2a7f6fe4e54\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"
Jul 28 01:26:40.722255 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.722077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"
Jul 28 01:26:40.862445 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:40.862415    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"]
Jul 28 01:26:40.866384 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:40.866359    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode937ea66_9f7b_4193_af0d_b2a7f6fe4e54.slice/crio-26ed3bfe76dd00f3ce9e3bffed959b3e71048b3d3e8080c8982b0d4155932227 WatchSource:0}: Error finding container 26ed3bfe76dd00f3ce9e3bffed959b3e71048b3d3e8080c8982b0d4155932227: Status 404 returned error can't find the container with id 26ed3bfe76dd00f3ce9e3bffed959b3e71048b3d3e8080c8982b0d4155932227
Jul 28 01:26:41.517729 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:41.517687    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4wn9f" event={"ID":"e2fe9fdd-797b-4c19-91c2-91218d0bf2ca","Type":"ContainerStarted","Data":"29e54e90db61ee39d28d4774c8b816223294cce47e09abefb71425ce224ed5a4"}
Jul 28 01:26:41.519239 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:41.519214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m789s" event={"ID":"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937","Type":"ContainerStarted","Data":"fb7be525f4bed34632d8c497c3a0535e333300c9d166094876649daabef54c86"}
Jul 28 01:26:41.520536 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:41.520509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4" event={"ID":"e937ea66-9f7b-4193-af0d-b2a7f6fe4e54","Type":"ContainerStarted","Data":"26ed3bfe76dd00f3ce9e3bffed959b3e71048b3d3e8080c8982b0d4155932227"}
Jul 28 01:26:41.523066 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:41.523046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbrls" event={"ID":"227a00b4-064f-4fef-af22-0a95f22a113c","Type":"ContainerStarted","Data":"c94b9fe8630c103e1e0e8bc0df992000b2a3b9dfc4f941088b8b2164c8bcb778"}
Jul 28 01:26:41.523152 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:41.523080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pbrls" event={"ID":"227a00b4-064f-4fef-af22-0a95f22a113c","Type":"ContainerStarted","Data":"b8f808ba2cbcb39b19dbdbc2bf8455da889d8824fcf86b37be61080149690509"}
Jul 28 01:26:41.523207 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:41.523194    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:41.535024 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:41.534973    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4wn9f" podStartSLOduration=1.764861642 podStartE2EDuration="7.534957709s" totalImagesPullingTime="5.770096067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:41.534752576 +0000 UTC m=+60.841241358" watchObservedRunningTime="2026-07-28 01:26:41.534957709 +0000 UTC m=+60.841446489"
Jul 28 01:26:41.556388 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:41.556337    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pbrls" podStartSLOduration=1.653407012 podStartE2EDuration="7.556319012s" totalImagesPullingTime="5.902912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:41.555156532 +0000 UTC m=+60.861645314" watchObservedRunningTime="2026-07-28 01:26:41.556319012 +0000 UTC m=+60.862807793"
Jul 28 01:26:44.535001 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:44.534907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m789s" event={"ID":"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937","Type":"ContainerStarted","Data":"6cf1c8c7f9585868e4572e01ef3ce7660029197ef60c92c117691770c56c614b"}
Jul 28 01:26:44.536307 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:44.536286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4" event={"ID":"e937ea66-9f7b-4193-af0d-b2a7f6fe4e54","Type":"ContainerStarted","Data":"0dab73be378d2ec7f0448fc8c244a74e9f09203d7303393d19a5759174afde33"}
Jul 28 01:26:44.536576 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:44.536550    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"
Jul 28 01:26:44.541353 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:44.541331    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4"
Jul 28 01:26:44.551185 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:44.551138    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9hcc4" podStartSLOduration=1.286982345 podStartE2EDuration="4.551108074s" totalImagesPullingTime="3.264125729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:44.551039768 +0000 UTC m=+63.857528551" watchObservedRunningTime="2026-07-28 01:26:44.551108074 +0000 UTC m=+63.857596856"
Jul 28 01:26:45.954726 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:45.954578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:45.954726 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:45.954669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:45.956959 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:45.956933    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:45.957090 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:45.956963    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:45.967495 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:45.967455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6f1795-9275-4b38-ba50-f2653f2b58dc-metrics-certs\") pod \"network-metrics-daemon-xskmq\" (UID: \"1c6f1795-9275-4b38-ba50-f2653f2b58dc\") " pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:45.967597 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:45.967526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021-original-pull-secret\") pod \"global-pull-secret-syncer-gd4x8\" (UID: \"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021\") " pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:45.973591 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:45.973571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gd4x8"
Jul 28 01:26:46.054979 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:46.054947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:46.057371 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:46.057351    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:46.067471 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:46.067442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:46.077983 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:46.077957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8s2h\" (UniqueName: \"kubernetes.io/projected/f88f1a31-21c3-45c5-8cc3-f44b91806734-kube-api-access-t8s2h\") pod \"network-check-target-2x9gm\" (UID: \"f88f1a31-21c3-45c5-8cc3-f44b91806734\") " pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:46.170457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:46.170423    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8db89\""
Jul 28 01:26:46.178336 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:46.178312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xskmq"
Jul 28 01:26:46.272607 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:46.272530    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4lwsn\""
Jul 28 01:26:46.280646 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:46.280619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:47.684377 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:47.684307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gd4x8"]
Jul 28 01:26:47.715630 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:47.715604    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xskmq"]
Jul 28 01:26:47.718338 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:47.718298    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6f1795_9275_4b38_ba50_f2653f2b58dc.slice/crio-865bc7e5547bc29ab6a2c8a6f100a13e3561bb55ceedd6a456c2500146d363a6 WatchSource:0}: Error finding container 865bc7e5547bc29ab6a2c8a6f100a13e3561bb55ceedd6a456c2500146d363a6: Status 404 returned error can't find the container with id 865bc7e5547bc29ab6a2c8a6f100a13e3561bb55ceedd6a456c2500146d363a6
Jul 28 01:26:47.734426 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:47.734407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2x9gm"]
Jul 28 01:26:47.737217 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:47.737188    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88f1a31_21c3_45c5_8cc3_f44b91806734.slice/crio-986ae0c9694e9c37400652a9a7afd8d4ce25899632053677f55b01f7f25f8ded WatchSource:0}: Error finding container 986ae0c9694e9c37400652a9a7afd8d4ce25899632053677f55b01f7f25f8ded: Status 404 returned error can't find the container with id 986ae0c9694e9c37400652a9a7afd8d4ce25899632053677f55b01f7f25f8ded
Jul 28 01:26:48.554583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:48.554514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gd4x8" event={"ID":"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021","Type":"ContainerStarted","Data":"92c5d4fc628777bc63d1e55a028b92acc45502b180d529e37bd35c730d634d9e"}
Jul 28 01:26:48.557105 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:48.557041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2x9gm" event={"ID":"f88f1a31-21c3-45c5-8cc3-f44b91806734","Type":"ContainerStarted","Data":"986ae0c9694e9c37400652a9a7afd8d4ce25899632053677f55b01f7f25f8ded"}
Jul 28 01:26:48.561205 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:48.561175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m789s" event={"ID":"6098fd8c-fce4-47e1-a5a1-bd9a75d1f937","Type":"ContainerStarted","Data":"1604ec943edc91435c00c1158c9ca184d53524e047911240dc0e3fd65c705522"}
Jul 28 01:26:48.563124 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:48.563098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xskmq" event={"ID":"1c6f1795-9275-4b38-ba50-f2653f2b58dc","Type":"ContainerStarted","Data":"865bc7e5547bc29ab6a2c8a6f100a13e3561bb55ceedd6a456c2500146d363a6"}
Jul 28 01:26:48.582770 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:48.582722    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m789s" podStartSLOduration=1.905166363 podStartE2EDuration="14.582703896s" totalImagesPullingTime="12.677537533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:48.581383909 +0000 UTC m=+67.887872694" watchObservedRunningTime="2026-07-28 01:26:48.582703896 +0000 UTC m=+67.889192680"
Jul 28 01:26:49.569563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:49.569457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xskmq" event={"ID":"1c6f1795-9275-4b38-ba50-f2653f2b58dc","Type":"ContainerStarted","Data":"01a0ddb0736480648579a14020ef1b9562f10c12ee68c995b2841d37c080b641"}
Jul 28 01:26:49.569563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:49.569548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xskmq" event={"ID":"1c6f1795-9275-4b38-ba50-f2653f2b58dc","Type":"ContainerStarted","Data":"cc57408f5f9d0a6ce9165c6a422c9a8a3ddbeba9b067ed2eb2432d98002393d5"}
Jul 28 01:26:49.600312 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:49.600250    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xskmq" podStartSLOduration=67.321811167 podStartE2EDuration="1m8.60022867s" totalImagesPullingTime="1.278417503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:49.5986945 +0000 UTC m=+68.905183280" watchObservedRunningTime="2026-07-28 01:26:49.60022867 +0000 UTC m=+68.906717455"
Jul 28 01:26:51.529533 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:51.529498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pbrls"
Jul 28 01:26:52.588444 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:52.588409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gd4x8" event={"ID":"fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021","Type":"ContainerStarted","Data":"2c3a85c67066876e77504ea5eddf4b639b99974f861e144df563d4a27badb999"}
Jul 28 01:26:52.590038 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:52.590007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2x9gm" event={"ID":"f88f1a31-21c3-45c5-8cc3-f44b91806734","Type":"ContainerStarted","Data":"4422f0dc259206195a5da9c70776c99fdf441b2570560aaa66705368b0ccc488"}
Jul 28 01:26:52.590148 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:52.590070    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:26:52.623381 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:52.623336    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gd4x8" podStartSLOduration=66.933524073 podStartE2EDuration="1m11.623323373s" totalImagesPullingTime="4.6897993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:52.622757119 +0000 UTC m=+71.929245900" watchObservedRunningTime="2026-07-28 01:26:52.623323373 +0000 UTC m=+71.929812163"
Jul 28 01:26:52.662846 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:52.662763    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2x9gm" podStartSLOduration=67.023491444 podStartE2EDuration="1m11.662750069s" totalImagesPullingTime="4.639258625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:52.661202897 +0000 UTC m=+71.967691678" watchObservedRunningTime="2026-07-28 01:26:52.662750069 +0000 UTC m=+71.969238850"
Jul 28 01:26:53.179804 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.179771    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ndmc2"]
Jul 28 01:26:53.181701 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.181685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.206508 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.206473    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 01:26:53.206626 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.206540    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 01:26:53.207906 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.207888    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 01:26:53.208001 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.207910    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 01:26:53.208058 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.208008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 01:26:53.210305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.210288    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mgxfm\""
Jul 28 01:26:53.212772 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.212759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 01:26:53.313686 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-textfile\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.313686 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.313935 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-wtmp\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.313935 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.313935 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313784    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-root\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.313935 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0852532d-b769-4dbd-a674-0e2472864fbc-metrics-client-ca\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.313935 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-tls\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.314096 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jn6\" (UniqueName: \"kubernetes.io/projected/0852532d-b769-4dbd-a674-0e2472864fbc-kube-api-access-g7jn6\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.314096 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.313993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-sys\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.414964 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.414930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-sys\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.414975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-textfile\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.414999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-wtmp\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-sys\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-root\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-root\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0852532d-b769-4dbd-a674-0e2472864fbc-metrics-client-ca\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415161    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-tls\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-wtmp\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jn6\" (UniqueName: \"kubernetes.io/projected/0852532d-b769-4dbd-a674-0e2472864fbc-kube-api-access-g7jn6\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-textfile\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415751 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0852532d-b769-4dbd-a674-0e2472864fbc-metrics-client-ca\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.415799 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.415744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-accelerators-collector-config\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.417391 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.417369    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-tls\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.417514 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.417394    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0852532d-b769-4dbd-a674-0e2472864fbc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.427999 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.427976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jn6\" (UniqueName: \"kubernetes.io/projected/0852532d-b769-4dbd-a674-0e2472864fbc-kube-api-access-g7jn6\") pod \"node-exporter-ndmc2\" (UID: \"0852532d-b769-4dbd-a674-0e2472864fbc\") " pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.490283 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.490225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ndmc2"
Jul 28 01:26:53.499581 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:53.499557    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0852532d_b769_4dbd_a674_0e2472864fbc.slice/crio-f1246db0c509572ba72480f74c71617a23fff9f2992470d7ca0e9b4f43c64708 WatchSource:0}: Error finding container f1246db0c509572ba72480f74c71617a23fff9f2992470d7ca0e9b4f43c64708: Status 404 returned error can't find the container with id f1246db0c509572ba72480f74c71617a23fff9f2992470d7ca0e9b4f43c64708
Jul 28 01:26:53.594569 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:53.594538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndmc2" event={"ID":"0852532d-b769-4dbd-a674-0e2472864fbc","Type":"ContainerStarted","Data":"f1246db0c509572ba72480f74c71617a23fff9f2992470d7ca0e9b4f43c64708"}
Jul 28 01:26:54.225969 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.225939    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:26:54.229148 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.229126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.232793 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.232767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 01:26:54.233412 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.233394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 01:26:54.234224 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.234196    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 01:26:54.234759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.234738    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 01:26:54.234759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.234751    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 01:26:54.234902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.234756    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 01:26:54.235198 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.235176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-28q42\""
Jul 28 01:26:54.235752 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.235731    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 01:26:54.242870 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.242842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 01:26:54.246752 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.246732    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 01:26:54.321288 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-config-out\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-web-config\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgm6\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-kube-api-access-qqgm6\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.321709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.321640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.422974 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.422942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.422974 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.422977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423189 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423189 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423189 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423189 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423189 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423189 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-config-out\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423214    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-web-config\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.423666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgm6\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-kube-api-access-qqgm6\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.425117 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.423971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.425117 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.424678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.425117 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.424756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.425658 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:54.425638    2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 28 01:26:54.425743 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:26:54.425701    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls podName:44145776-6669-4725-8d4b-17aa70cd40ce nodeName:}" failed. No retries permitted until 2026-07-28 01:26:54.925682786 +0000 UTC m=+74.232171559 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce") : secret "alertmanager-main-tls" not found
Jul 28 01:26:54.425743 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.425726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.426753 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.426728    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:26:54.426852 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.426728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.428135 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.428096    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.428685 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.428634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.428790 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.428684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.428933 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.428907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-config-out\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.429819 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.429799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-web-config\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.430690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.430652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.456339 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.456302    2579 patch_prober.go:28] interesting pod/image-registry-c6d86cc9-69rg5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 01:26:54.456541 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.456361    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-c6d86cc9-69rg5" podUID="a4498e9e-3eff-445b-b3af-23d0ad786c73" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:26:54.457634 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.457610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgm6\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-kube-api-access-qqgm6\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.603256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.603213    2579 generic.go:356] "Generic (PLEG): container finished" podID="0852532d-b769-4dbd-a674-0e2472864fbc" containerID="da38a811eb2579d0262f185165ad4d280651385b80e836dff3ede11dfe8992fb" exitCode=0
Jul 28 01:26:54.603703 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.603292    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndmc2" event={"ID":"0852532d-b769-4dbd-a674-0e2472864fbc","Type":"ContainerDied","Data":"da38a811eb2579d0262f185165ad4d280651385b80e836dff3ede11dfe8992fb"}
Jul 28 01:26:54.926495 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.926445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:54.928759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:54.928731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:55.140954 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:55.140920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:26:55.278991 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:55.278842    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:26:55.281886 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:55.281856    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44145776_6669_4725_8d4b_17aa70cd40ce.slice/crio-126986446d40d780f746aba7a88593c635e894e4c63818453ac31b9fd919513c WatchSource:0}: Error finding container 126986446d40d780f746aba7a88593c635e894e4c63818453ac31b9fd919513c: Status 404 returned error can't find the container with id 126986446d40d780f746aba7a88593c635e894e4c63818453ac31b9fd919513c
Jul 28 01:26:55.608305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:55.608222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndmc2" event={"ID":"0852532d-b769-4dbd-a674-0e2472864fbc","Type":"ContainerStarted","Data":"a461f0aa9a8afa2eda4434011b76ba40f3fa5850daac1850fe3543af50fe333c"}
Jul 28 01:26:55.608305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:55.608276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ndmc2" event={"ID":"0852532d-b769-4dbd-a674-0e2472864fbc","Type":"ContainerStarted","Data":"1820f98fb6f10340dfdd664a6032b9633b3c34661b52f0fe7e063931ff60fb97"}
Jul 28 01:26:55.609368 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:55.609346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerStarted","Data":"126986446d40d780f746aba7a88593c635e894e4c63818453ac31b9fd919513c"}
Jul 28 01:26:56.107905 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.107461    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ndmc2" podStartSLOduration=2.182236883 podStartE2EDuration="3.107438673s" totalImagesPullingTime="834.198394ms" totalInitContainerRuntime="91.003396ms" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:55.635741008 +0000 UTC m=+74.942229792" watchObservedRunningTime="2026-07-28 01:26:56.107438673 +0000 UTC m=+75.413927456"
Jul 28 01:26:56.108605 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.108587    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"]
Jul 28 01:26:56.111307 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.111279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.114025 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.114000    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 01:26:56.114140 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.114048    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6c2obrqitn1h\""
Jul 28 01:26:56.114140 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.114069    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 01:26:56.114362 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.114345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 01:26:56.114667 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.114649    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 01:26:56.114736 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.114701    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2ckkn\""
Jul 28 01:26:56.114789 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.114774    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 01:26:56.135350 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.135324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"]
Jul 28 01:26:56.237476 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.237441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.237476 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.237478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fd27f23-16b3-4a2c-8e52-521bb3be8940-metrics-client-ca\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.237732 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.237519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7j54\" (UniqueName: \"kubernetes.io/projected/7fd27f23-16b3-4a2c-8e52-521bb3be8940-kube-api-access-k7j54\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.237732 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.237589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.237732 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.237644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.237732 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.237682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-grpc-tls\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.237937 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.237749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-tls\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.237937 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.237793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.338780 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.338743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.338780 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.338781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fd27f23-16b3-4a2c-8e52-521bb3be8940-metrics-client-ca\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.339000 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.338803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7j54\" (UniqueName: \"kubernetes.io/projected/7fd27f23-16b3-4a2c-8e52-521bb3be8940-kube-api-access-k7j54\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.339000 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.338849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.339000 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.338876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.339000 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.338914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-grpc-tls\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.339000 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.338946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-tls\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.339000 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.338974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.339690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.339638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7fd27f23-16b3-4a2c-8e52-521bb3be8940-metrics-client-ca\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.341407 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.341382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.341598 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.341508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.341936 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.341909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-tls\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.342008 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.341919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-grpc-tls\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.342008 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.341954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.342244 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.342227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7fd27f23-16b3-4a2c-8e52-521bb3be8940-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.355712 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.355687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7j54\" (UniqueName: \"kubernetes.io/projected/7fd27f23-16b3-4a2c-8e52-521bb3be8940-kube-api-access-k7j54\") pod \"thanos-querier-57bd896dcd-9zcrd\" (UID: \"7fd27f23-16b3-4a2c-8e52-521bb3be8940\") " pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.422472 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.422439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:26:56.568451 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.568415    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"]
Jul 28 01:26:56.571902 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:56.571875    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd27f23_16b3_4a2c_8e52_521bb3be8940.slice/crio-702e99c7a348e087465cf797ed6351d57877f7747c5de6d2eabf93b1a750d9da WatchSource:0}: Error finding container 702e99c7a348e087465cf797ed6351d57877f7747c5de6d2eabf93b1a750d9da: Status 404 returned error can't find the container with id 702e99c7a348e087465cf797ed6351d57877f7747c5de6d2eabf93b1a750d9da
Jul 28 01:26:56.613793 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.613759    2579 generic.go:356] "Generic (PLEG): container finished" podID="44145776-6669-4725-8d4b-17aa70cd40ce" containerID="6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c" exitCode=0
Jul 28 01:26:56.614244 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.613833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerDied","Data":"6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c"}
Jul 28 01:26:56.615116 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:56.615091    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd" event={"ID":"7fd27f23-16b3-4a2c-8e52-521bb3be8940","Type":"ContainerStarted","Data":"702e99c7a348e087465cf797ed6351d57877f7747c5de6d2eabf93b1a750d9da"}
Jul 28 01:26:57.509374 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.509334    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c6d86cc9-69rg5"
Jul 28 01:26:57.770152 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.770062    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-79c8557f5c-759hx"]
Jul 28 01:26:57.772527 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.772497    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.776359 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.776203    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 01:26:57.776761 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.776735    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bi24ab66uoc8v\""
Jul 28 01:26:57.776895 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.776811    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 01:26:57.776966 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.776947    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 01:26:57.777023 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.776998    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 01:26:57.777109 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.776947    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-26knf\""
Jul 28 01:26:57.787039 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.787018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79c8557f5c-759hx"]
Jul 28 01:26:57.851504 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.851455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-secret-metrics-server-client-certs\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.851661 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.851510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrzh\" (UniqueName: \"kubernetes.io/projected/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-kube-api-access-4nrzh\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.851661 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.851539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-audit-log\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.851661 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.851567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.851661 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.851590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-client-ca-bundle\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.851882 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.851746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-secret-metrics-server-tls\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.851882 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.851793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-metrics-server-audit-profiles\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.952640 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.952603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-secret-metrics-server-tls\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.952816 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.952655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-metrics-server-audit-profiles\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.952816 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.952701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-secret-metrics-server-client-certs\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.952816 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.952729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrzh\" (UniqueName: \"kubernetes.io/projected/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-kube-api-access-4nrzh\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.952816 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.952755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-audit-log\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.952816 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.952786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.953071 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.952819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-client-ca-bundle\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.953680 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.953439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-audit-log\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.954012 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.953962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.954186 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.954162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-metrics-server-audit-profiles\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.955699 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.955677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-client-ca-bundle\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.955809 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.955754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-secret-metrics-server-tls\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.956152 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.956130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-secret-metrics-server-client-certs\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:57.963021 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:57.963003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrzh\" (UniqueName: \"kubernetes.io/projected/d19fecca-b180-4bbf-a4b4-dd38ffcd0338-kube-api-access-4nrzh\") pod \"metrics-server-79c8557f5c-759hx\" (UID: \"d19fecca-b180-4bbf-a4b4-dd38ffcd0338\") " pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:58.084743 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:58.084670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:26:58.884312 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:58.883894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-79c8557f5c-759hx"]
Jul 28 01:26:58.887809 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:26:58.887767    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19fecca_b180_4bbf_a4b4_dd38ffcd0338.slice/crio-3126817e555dbc7fa7ca356b6f3ac9e964024c8d4066ad08edef872e4b1e6ce7 WatchSource:0}: Error finding container 3126817e555dbc7fa7ca356b6f3ac9e964024c8d4066ad08edef872e4b1e6ce7: Status 404 returned error can't find the container with id 3126817e555dbc7fa7ca356b6f3ac9e964024c8d4066ad08edef872e4b1e6ce7
Jul 28 01:26:59.630204 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.630144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx" event={"ID":"d19fecca-b180-4bbf-a4b4-dd38ffcd0338","Type":"ContainerStarted","Data":"3126817e555dbc7fa7ca356b6f3ac9e964024c8d4066ad08edef872e4b1e6ce7"}
Jul 28 01:26:59.634261 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.634229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerStarted","Data":"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4"}
Jul 28 01:26:59.634395 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.634305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerStarted","Data":"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b"}
Jul 28 01:26:59.634395 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.634347    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerStarted","Data":"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c"}
Jul 28 01:26:59.634395 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.634384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerStarted","Data":"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6"}
Jul 28 01:26:59.634571 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.634421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerStarted","Data":"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62"}
Jul 28 01:26:59.637133 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.637106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd" event={"ID":"7fd27f23-16b3-4a2c-8e52-521bb3be8940","Type":"ContainerStarted","Data":"32786560c9d333776685efac471847b39466c86160da8c4d95d8bb8a6a90f654"}
Jul 28 01:26:59.637265 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.637177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd" event={"ID":"7fd27f23-16b3-4a2c-8e52-521bb3be8940","Type":"ContainerStarted","Data":"dab19ad2cb68846c5be4246531cfa33b94c6ae9734d2f0007d6b46bb9d479fb0"}
Jul 28 01:26:59.637265 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:26:59.637218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd" event={"ID":"7fd27f23-16b3-4a2c-8e52-521bb3be8940","Type":"ContainerStarted","Data":"e9830a8cf715898850df70916b773d54029f6bed27daea8fb1c34f3eb36007a9"}
Jul 28 01:27:00.184615 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.184582    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d8dc675f5-s6jp7"]
Jul 28 01:27:00.187150 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.187128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.189665 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.189642    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 01:27:00.190589 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.190415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 01:27:00.190589 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.190427    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 01:27:00.190589 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.190444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 01:27:00.190589 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.190469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 01:27:00.190589 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.190422    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 01:27:00.190921 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.190622    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 01:27:00.190921 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.190882    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7m25v\""
Jul 28 01:27:00.194912 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.194891    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 01:27:00.202879 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.202856    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d8dc675f5-s6jp7"]
Jul 28 01:27:00.271033 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.271011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-serving-cert\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.271140 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.271048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-oauth-serving-cert\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.271140 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.271075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-trusted-ca-bundle\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.271226 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.271179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-oauth-config\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.271274 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.271251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-config\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.271307 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.271296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clc5f\" (UniqueName: \"kubernetes.io/projected/97476fe7-be95-42c8-bbc8-449b21b8d5c8-kube-api-access-clc5f\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.271347 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.271338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-service-ca\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.372731 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.372709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-service-ca\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.372816 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.372753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-serving-cert\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.372816 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.372785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-oauth-serving-cert\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.372926 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.372821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-trusted-ca-bundle\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.373457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.373041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-oauth-config\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.373457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.373100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-config\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.373457 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.373128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clc5f\" (UniqueName: \"kubernetes.io/projected/97476fe7-be95-42c8-bbc8-449b21b8d5c8-kube-api-access-clc5f\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.373675 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.373546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-service-ca\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.373675 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.373590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-oauth-serving-cert\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.373675 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.373633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-trusted-ca-bundle\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.373962 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.373941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-config\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.375344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.375327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-serving-cert\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.375775 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.375758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-oauth-config\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.390165 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.390144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clc5f\" (UniqueName: \"kubernetes.io/projected/97476fe7-be95-42c8-bbc8-449b21b8d5c8-kube-api-access-clc5f\") pod \"console-d8dc675f5-s6jp7\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") " pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.498208 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.498125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:00.624729 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.624698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d8dc675f5-s6jp7"]
Jul 28 01:27:00.627515 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:27:00.627469    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97476fe7_be95_42c8_bbc8_449b21b8d5c8.slice/crio-f0647cd082f5209cf131d4fbcfd1d6e73e132dcbe808370485949feedb8b322c WatchSource:0}: Error finding container f0647cd082f5209cf131d4fbcfd1d6e73e132dcbe808370485949feedb8b322c: Status 404 returned error can't find the container with id f0647cd082f5209cf131d4fbcfd1d6e73e132dcbe808370485949feedb8b322c
Jul 28 01:27:00.643580 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.643544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerStarted","Data":"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1"}
Jul 28 01:27:00.646658 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.646627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd" event={"ID":"7fd27f23-16b3-4a2c-8e52-521bb3be8940","Type":"ContainerStarted","Data":"e1297227940eeafd094b8c8fc965be867697ddcd3268d8556ee3cdec8ddada3c"}
Jul 28 01:27:00.646767 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.646695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd" event={"ID":"7fd27f23-16b3-4a2c-8e52-521bb3be8940","Type":"ContainerStarted","Data":"ec0498d7b03edafc6e6da74054f5d01866a7113738a2593229285ddaabe93cff"}
Jul 28 01:27:00.646767 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.646741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:27:00.646878 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.646775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd" event={"ID":"7fd27f23-16b3-4a2c-8e52-521bb3be8940","Type":"ContainerStarted","Data":"526fb1b1f54485ec419148309eb37187203b523426a3d625fdf64cc193d4fcf6"}
Jul 28 01:27:00.648583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.648561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx" event={"ID":"d19fecca-b180-4bbf-a4b4-dd38ffcd0338","Type":"ContainerStarted","Data":"432e9c62088cf48302df626695fa52072de2c9a82c16d422ef1a34a4c0edbd65"}
Jul 28 01:27:00.649640 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.649621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8dc675f5-s6jp7" event={"ID":"97476fe7-be95-42c8-bbc8-449b21b8d5c8","Type":"ContainerStarted","Data":"f0647cd082f5209cf131d4fbcfd1d6e73e132dcbe808370485949feedb8b322c"}
Jul 28 01:27:00.674188 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.674129    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.687822906 podStartE2EDuration="6.674115827s" totalImagesPullingTime="3.965833498s" totalInitContainerRuntime="20.459423ms" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:00.673251858 +0000 UTC m=+79.979740638" watchObservedRunningTime="2026-07-28 01:27:00.674115827 +0000 UTC m=+79.980604608"
Jul 28 01:27:00.711065 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.711017    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd" podStartSLOduration=1.665089278 podStartE2EDuration="4.711001403s" totalImagesPullingTime="3.045912125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:00.707803618 +0000 UTC m=+80.014292400" watchObservedRunningTime="2026-07-28 01:27:00.711001403 +0000 UTC m=+80.017490184"
Jul 28 01:27:00.735260 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:00.735209    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx" podStartSLOduration=2.376408567 podStartE2EDuration="3.735191617s" totalImagesPullingTime="1.35878305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:00.733863462 +0000 UTC m=+80.040352293" watchObservedRunningTime="2026-07-28 01:27:00.735191617 +0000 UTC m=+80.041680398"
Jul 28 01:27:03.664061 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:03.664030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8dc675f5-s6jp7" event={"ID":"97476fe7-be95-42c8-bbc8-449b21b8d5c8","Type":"ContainerStarted","Data":"b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45"}
Jul 28 01:27:03.684538 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:03.684475    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d8dc675f5-s6jp7" podStartSLOduration=1.001073206 podStartE2EDuration="3.684460932s" totalImagesPullingTime="2.683387726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:03.682774987 +0000 UTC m=+82.989263768" watchObservedRunningTime="2026-07-28 01:27:03.684460932 +0000 UTC m=+82.990949713"
Jul 28 01:27:06.661434 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:06.661404    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57bd896dcd-9zcrd"
Jul 28 01:27:10.498464 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:10.498421    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:10.499012 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:10.498672    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:10.503502 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:10.503469    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:10.693355 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:10.693328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:17.237440 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.237407    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c75fbb856-gx8kp"]
Jul 28 01:27:17.527413 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.527340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c75fbb856-gx8kp"]
Jul 28 01:27:17.527572 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.527433    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.613214 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.613182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-trusted-ca-bundle\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.613214 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.613219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-service-ca\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.613420 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.613237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-oauth-serving-cert\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.613420 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.613256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-config\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.613420 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.613313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-oauth-config\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.613420 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.613373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsbf\" (UniqueName: \"kubernetes.io/projected/71055f96-9be9-4dcf-95f9-b16b1abfb65f-kube-api-access-gqsbf\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.613420 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.613408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-serving-cert\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.713936 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.713904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsbf\" (UniqueName: \"kubernetes.io/projected/71055f96-9be9-4dcf-95f9-b16b1abfb65f-kube-api-access-gqsbf\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.714120 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.713944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-serving-cert\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.714120 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.713977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-trusted-ca-bundle\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.714120 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.714028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-service-ca\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.714120 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.714067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-oauth-serving-cert\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.714120 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.714100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-config\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.714381 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.714129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-oauth-config\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.714921 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.714897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-trusted-ca-bundle\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.715048 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.714940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-config\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.715048 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.714916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-oauth-serving-cert\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.715254 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.715237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-service-ca\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.716598 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.716577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-serving-cert\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.716754 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.716739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-oauth-config\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.722213 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.722192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsbf\" (UniqueName: \"kubernetes.io/projected/71055f96-9be9-4dcf-95f9-b16b1abfb65f-kube-api-access-gqsbf\") pod \"console-5c75fbb856-gx8kp\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") " pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.836391 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.836313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:17.996276 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:17.996245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c75fbb856-gx8kp"]
Jul 28 01:27:18.000049 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:27:18.000008    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71055f96_9be9_4dcf_95f9_b16b1abfb65f.slice/crio-a8429442dacc45fef3cf6ee030d41e3e53886c02db172358288d4a41fd7706f5 WatchSource:0}: Error finding container a8429442dacc45fef3cf6ee030d41e3e53886c02db172358288d4a41fd7706f5: Status 404 returned error can't find the container with id a8429442dacc45fef3cf6ee030d41e3e53886c02db172358288d4a41fd7706f5
Jul 28 01:27:18.085331 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:18.085304    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:27:18.085434 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:18.085363    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:27:18.725815 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:18.725778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c75fbb856-gx8kp" event={"ID":"71055f96-9be9-4dcf-95f9-b16b1abfb65f","Type":"ContainerStarted","Data":"d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32"}
Jul 28 01:27:18.726192 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:18.725861    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c75fbb856-gx8kp" event={"ID":"71055f96-9be9-4dcf-95f9-b16b1abfb65f","Type":"ContainerStarted","Data":"a8429442dacc45fef3cf6ee030d41e3e53886c02db172358288d4a41fd7706f5"}
Jul 28 01:27:18.746689 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:18.746625    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c75fbb856-gx8kp" podStartSLOduration=1.746608514 podStartE2EDuration="1.746608514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:18.745603985 +0000 UTC m=+98.052092785" watchObservedRunningTime="2026-07-28 01:27:18.746608514 +0000 UTC m=+98.053097294"
Jul 28 01:27:20.319958 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:20.319928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4wn9f_e2fe9fdd-797b-4c19-91c2-91218d0bf2ca/serve-healthcheck-canary/0.log"
Jul 28 01:27:23.596941 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:23.596915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2x9gm"
Jul 28 01:27:27.836421 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:27.836393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:27.836811 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:27.836471    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:27.841264 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:27.841243    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:28.763766 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:28.763740    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:27:28.830508 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:28.830466    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d8dc675f5-s6jp7"]
Jul 28 01:27:38.090672 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:38.090641    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:27:38.094406 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:38.094376    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-79c8557f5c-759hx"
Jul 28 01:27:53.853192 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:53.853143    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d8dc675f5-s6jp7" podUID="97476fe7-be95-42c8-bbc8-449b21b8d5c8" containerName="console" containerID="cri-o://b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45" gracePeriod=15
Jul 28 01:27:54.101074 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.101051    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:54.231176 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231141    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clc5f\" (UniqueName: \"kubernetes.io/projected/97476fe7-be95-42c8-bbc8-449b21b8d5c8-kube-api-access-clc5f\") pod \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") "
Jul 28 01:27:54.231453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-oauth-config\") pod \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") "
Jul 28 01:27:54.231453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231233    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-serving-cert\") pod \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") "
Jul 28 01:27:54.231453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-config\") pod \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") "
Jul 28 01:27:54.231453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-trusted-ca-bundle\") pod \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") "
Jul 28 01:27:54.231453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-service-ca\") pod \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") "
Jul 28 01:27:54.231453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-oauth-serving-cert\") pod \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\" (UID: \"97476fe7-be95-42c8-bbc8-449b21b8d5c8\") "
Jul 28 01:27:54.231798 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-config" (OuterVolumeSpecName: "console-config") pod "97476fe7-be95-42c8-bbc8-449b21b8d5c8" (UID: "97476fe7-be95-42c8-bbc8-449b21b8d5c8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:54.231861 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231788    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "97476fe7-be95-42c8-bbc8-449b21b8d5c8" (UID: "97476fe7-be95-42c8-bbc8-449b21b8d5c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:54.232015 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.231989    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-service-ca" (OuterVolumeSpecName: "service-ca") pod "97476fe7-be95-42c8-bbc8-449b21b8d5c8" (UID: "97476fe7-be95-42c8-bbc8-449b21b8d5c8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:54.232095 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.232010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97476fe7-be95-42c8-bbc8-449b21b8d5c8" (UID: "97476fe7-be95-42c8-bbc8-449b21b8d5c8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:54.233564 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.233534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97476fe7-be95-42c8-bbc8-449b21b8d5c8" (UID: "97476fe7-be95-42c8-bbc8-449b21b8d5c8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:54.233654 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.233594    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97476fe7-be95-42c8-bbc8-449b21b8d5c8-kube-api-access-clc5f" (OuterVolumeSpecName: "kube-api-access-clc5f") pod "97476fe7-be95-42c8-bbc8-449b21b8d5c8" (UID: "97476fe7-be95-42c8-bbc8-449b21b8d5c8"). InnerVolumeSpecName "kube-api-access-clc5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:54.233654 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.233616    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97476fe7-be95-42c8-bbc8-449b21b8d5c8" (UID: "97476fe7-be95-42c8-bbc8-449b21b8d5c8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:54.332146 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.332115    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-trusted-ca-bundle\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:27:54.332146 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.332140    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-service-ca\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:27:54.332146 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.332150    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-oauth-serving-cert\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:27:54.332376 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.332161    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clc5f\" (UniqueName: \"kubernetes.io/projected/97476fe7-be95-42c8-bbc8-449b21b8d5c8-kube-api-access-clc5f\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:27:54.332376 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.332170    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-oauth-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:27:54.332376 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.332180    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-serving-cert\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:27:54.332376 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.332188    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97476fe7-be95-42c8-bbc8-449b21b8d5c8-console-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:27:54.856434 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.856391    2579 generic.go:356] "Generic (PLEG): container finished" podID="97476fe7-be95-42c8-bbc8-449b21b8d5c8" containerID="b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45" exitCode=0
Jul 28 01:27:54.856873 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.856462    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8dc675f5-s6jp7"
Jul 28 01:27:54.856873 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.856507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8dc675f5-s6jp7" event={"ID":"97476fe7-be95-42c8-bbc8-449b21b8d5c8","Type":"ContainerDied","Data":"b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45"}
Jul 28 01:27:54.856873 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.856576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8dc675f5-s6jp7" event={"ID":"97476fe7-be95-42c8-bbc8-449b21b8d5c8","Type":"ContainerDied","Data":"f0647cd082f5209cf131d4fbcfd1d6e73e132dcbe808370485949feedb8b322c"}
Jul 28 01:27:54.856873 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.856597    2579 scope.go:117] "RemoveContainer" containerID="b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45"
Jul 28 01:27:54.865357 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.865339    2579 scope.go:117] "RemoveContainer" containerID="b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45"
Jul 28 01:27:54.865661 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:27:54.865639    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45\": container with ID starting with b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45 not found: ID does not exist" containerID="b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45"
Jul 28 01:27:54.865733 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.865670    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45"} err="failed to get container status \"b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45\": rpc error: code = NotFound desc = could not find container \"b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45\": container with ID starting with b04567f9f092b0f998edfaff3b8a3a27269379bfebb2cf3dfc7567d83186ea45 not found: ID does not exist"
Jul 28 01:27:54.877171 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.877145    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d8dc675f5-s6jp7"]
Jul 28 01:27:54.880768 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:54.880746    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d8dc675f5-s6jp7"]
Jul 28 01:27:55.162274 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:27:55.162233    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97476fe7-be95-42c8-bbc8-449b21b8d5c8" path="/var/lib/kubelet/pods/97476fe7-be95-42c8-bbc8-449b21b8d5c8/volumes"
Jul 28 01:28:13.467693 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.467661    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:28:13.468114 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.467923    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="alertmanager" containerID="cri-o://f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62" gracePeriod=120
Jul 28 01:28:13.468114 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.468005    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy-metric" containerID="cri-o://b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4" gracePeriod=120
Jul 28 01:28:13.468114 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.468032    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy-web" containerID="cri-o://4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c" gracePeriod=120
Jul 28 01:28:13.468232 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.468057    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="config-reloader" containerID="cri-o://9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6" gracePeriod=120
Jul 28 01:28:13.468443 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.468393    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="prom-label-proxy" containerID="cri-o://6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1" gracePeriod=120
Jul 28 01:28:13.468595 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.468502    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy" containerID="cri-o://25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b" gracePeriod=120
Jul 28 01:28:13.928145 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.928114    2579 generic.go:356] "Generic (PLEG): container finished" podID="44145776-6669-4725-8d4b-17aa70cd40ce" containerID="6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1" exitCode=0
Jul 28 01:28:13.928145 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.928138    2579 generic.go:356] "Generic (PLEG): container finished" podID="44145776-6669-4725-8d4b-17aa70cd40ce" containerID="25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b" exitCode=0
Jul 28 01:28:13.928145 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.928145    2579 generic.go:356] "Generic (PLEG): container finished" podID="44145776-6669-4725-8d4b-17aa70cd40ce" containerID="9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6" exitCode=0
Jul 28 01:28:13.928145 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.928151    2579 generic.go:356] "Generic (PLEG): container finished" podID="44145776-6669-4725-8d4b-17aa70cd40ce" containerID="f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62" exitCode=0
Jul 28 01:28:13.928395 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.928192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerDied","Data":"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1"}
Jul 28 01:28:13.928395 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.928282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerDied","Data":"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b"}
Jul 28 01:28:13.928395 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.928307    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerDied","Data":"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6"}
Jul 28 01:28:13.928395 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:13.928332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerDied","Data":"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62"}
Jul 28 01:28:14.708026 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.708004    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:14.788524 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788424    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.788524 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788474    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-config-out\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.788524 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788510    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-config-volume\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.788782 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-cluster-tls-config\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.788782 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqgm6\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-kube-api-access-qqgm6\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.788782 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788593    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.788782 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788618    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-main-db\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.788782 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788717    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.788782 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-metrics-client-ca\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.789117 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-web-config\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.789117 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788850    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.789117 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-trusted-ca-bundle\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.789117 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.788923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-tls-assets\") pod \"44145776-6669-4725-8d4b-17aa70cd40ce\" (UID: \"44145776-6669-4725-8d4b-17aa70cd40ce\") "
Jul 28 01:28:14.790654 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.790518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:14.790775 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.790727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:14.790855 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.790835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:28:14.791305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.791270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-config-out" (OuterVolumeSpecName: "config-out") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:28:14.791421 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.791395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:14.791512 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.791464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:14.791708 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.791612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:14.791708 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.791676    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:28:14.791960 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.791937    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-config-volume" (OuterVolumeSpecName: "config-volume") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:14.792182 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.792152    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-kube-api-access-qqgm6" (OuterVolumeSpecName: "kube-api-access-qqgm6") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "kube-api-access-qqgm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:28:14.793283 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.793251    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:14.796153 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.796131    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:14.801691 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.801671    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-web-config" (OuterVolumeSpecName: "web-config") pod "44145776-6669-4725-8d4b-17aa70cd40ce" (UID: "44145776-6669-4725-8d4b-17aa70cd40ce"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:14.889761 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889736    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-main-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889761 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889759    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-metrics-client-ca\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889769    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-web-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889779    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889788    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889797    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-tls-assets\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889806    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889814    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-config-out\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889823    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-config-volume\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889831    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-cluster-tls-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889839    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqgm6\" (UniqueName: \"kubernetes.io/projected/44145776-6669-4725-8d4b-17aa70cd40ce-kube-api-access-qqgm6\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889848    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/44145776-6669-4725-8d4b-17aa70cd40ce-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.889902 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.889856    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/44145776-6669-4725-8d4b-17aa70cd40ce-alertmanager-main-db\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:28:14.935036 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.935005    2579 generic.go:356] "Generic (PLEG): container finished" podID="44145776-6669-4725-8d4b-17aa70cd40ce" containerID="b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4" exitCode=0
Jul 28 01:28:14.935036 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.935030    2579 generic.go:356] "Generic (PLEG): container finished" podID="44145776-6669-4725-8d4b-17aa70cd40ce" containerID="4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c" exitCode=0
Jul 28 01:28:14.935172 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.935081    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:14.935172 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.935088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerDied","Data":"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4"}
Jul 28 01:28:14.935172 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.935159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerDied","Data":"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c"}
Jul 28 01:28:14.935265 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.935186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"44145776-6669-4725-8d4b-17aa70cd40ce","Type":"ContainerDied","Data":"126986446d40d780f746aba7a88593c635e894e4c63818453ac31b9fd919513c"}
Jul 28 01:28:14.935265 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.935225    2579 scope.go:117] "RemoveContainer" containerID="6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1"
Jul 28 01:28:14.942632 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.942614    2579 scope.go:117] "RemoveContainer" containerID="b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4"
Jul 28 01:28:14.949758 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.949741    2579 scope.go:117] "RemoveContainer" containerID="25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b"
Jul 28 01:28:14.956375 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.956356    2579 scope.go:117] "RemoveContainer" containerID="4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c"
Jul 28 01:28:14.958186 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.958165    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:28:14.962301 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.962279    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:28:14.963651 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.963636    2579 scope.go:117] "RemoveContainer" containerID="9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6"
Jul 28 01:28:14.970083 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.970065    2579 scope.go:117] "RemoveContainer" containerID="f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62"
Jul 28 01:28:14.977993 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.977976    2579 scope.go:117] "RemoveContainer" containerID="6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c"
Jul 28 01:28:14.984518 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.984501    2579 scope.go:117] "RemoveContainer" containerID="6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1"
Jul 28 01:28:14.984751 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:28:14.984734    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1\": container with ID starting with 6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1 not found: ID does not exist" containerID="6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1"
Jul 28 01:28:14.984802 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.984760    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1"} err="failed to get container status \"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1\": rpc error: code = NotFound desc = could not find container \"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1\": container with ID starting with 6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1 not found: ID does not exist"
Jul 28 01:28:14.984802 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.984778    2579 scope.go:117] "RemoveContainer" containerID="b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4"
Jul 28 01:28:14.985003 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:28:14.984985    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4\": container with ID starting with b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4 not found: ID does not exist" containerID="b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4"
Jul 28 01:28:14.985044 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.985011    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4"} err="failed to get container status \"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4\": rpc error: code = NotFound desc = could not find container \"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4\": container with ID starting with b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4 not found: ID does not exist"
Jul 28 01:28:14.985044 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.985028    2579 scope.go:117] "RemoveContainer" containerID="25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b"
Jul 28 01:28:14.985238 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:28:14.985220    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b\": container with ID starting with 25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b not found: ID does not exist" containerID="25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b"
Jul 28 01:28:14.985277 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.985242    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b"} err="failed to get container status \"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b\": rpc error: code = NotFound desc = could not find container \"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b\": container with ID starting with 25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b not found: ID does not exist"
Jul 28 01:28:14.985277 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.985256    2579 scope.go:117] "RemoveContainer" containerID="4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c"
Jul 28 01:28:14.985470 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:28:14.985456    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c\": container with ID starting with 4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c not found: ID does not exist" containerID="4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c"
Jul 28 01:28:14.985529 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.985472    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c"} err="failed to get container status \"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c\": rpc error: code = NotFound desc = could not find container \"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c\": container with ID starting with 4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c not found: ID does not exist"
Jul 28 01:28:14.985529 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.985526    2579 scope.go:117] "RemoveContainer" containerID="9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6"
Jul 28 01:28:14.985765 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:28:14.985748    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6\": container with ID starting with 9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6 not found: ID does not exist" containerID="9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6"
Jul 28 01:28:14.985806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.985769    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6"} err="failed to get container status \"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6\": rpc error: code = NotFound desc = could not find container \"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6\": container with ID starting with 9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6 not found: ID does not exist"
Jul 28 01:28:14.985806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.985782    2579 scope.go:117] "RemoveContainer" containerID="f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62"
Jul 28 01:28:14.986002 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:28:14.985981    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62\": container with ID starting with f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62 not found: ID does not exist" containerID="f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62"
Jul 28 01:28:14.986036 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986007    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62"} err="failed to get container status \"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62\": rpc error: code = NotFound desc = could not find container \"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62\": container with ID starting with f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62 not found: ID does not exist"
Jul 28 01:28:14.986036 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986020    2579 scope.go:117] "RemoveContainer" containerID="6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c"
Jul 28 01:28:14.986244 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:28:14.986227    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c\": container with ID starting with 6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c not found: ID does not exist" containerID="6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c"
Jul 28 01:28:14.986288 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986248    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c"} err="failed to get container status \"6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c\": rpc error: code = NotFound desc = could not find container \"6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c\": container with ID starting with 6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c not found: ID does not exist"
Jul 28 01:28:14.986288 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986262    2579 scope.go:117] "RemoveContainer" containerID="6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1"
Jul 28 01:28:14.986520 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986477    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1"} err="failed to get container status \"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1\": rpc error: code = NotFound desc = could not find container \"6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1\": container with ID starting with 6717883a4ba55503c855ae406a58b0539aeefff6384fda5315a69b38add3daf1 not found: ID does not exist"
Jul 28 01:28:14.986612 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986521    2579 scope.go:117] "RemoveContainer" containerID="b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4"
Jul 28 01:28:14.986753 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986733    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4"} err="failed to get container status \"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4\": rpc error: code = NotFound desc = could not find container \"b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4\": container with ID starting with b5eb617451933bf05212d85398751d313e293e20668b1ea3ae31e83b134d7dc4 not found: ID does not exist"
Jul 28 01:28:14.986823 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986756    2579 scope.go:117] "RemoveContainer" containerID="25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b"
Jul 28 01:28:14.986935 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986918    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b"} err="failed to get container status \"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b\": rpc error: code = NotFound desc = could not find container \"25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b\": container with ID starting with 25f5e5e3193c5ac1b20cca23d45b1f2f8776f6359c520df67b5e07a58d879a9b not found: ID does not exist"
Jul 28 01:28:14.986975 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.986937    2579 scope.go:117] "RemoveContainer" containerID="4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c"
Jul 28 01:28:14.987154 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.987138    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c"} err="failed to get container status \"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c\": rpc error: code = NotFound desc = could not find container \"4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c\": container with ID starting with 4936d343c4b88fa7158725904f24d30e0195616a47ef2ef5280d5b03d4ffaa8c not found: ID does not exist"
Jul 28 01:28:14.987220 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.987156    2579 scope.go:117] "RemoveContainer" containerID="9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6"
Jul 28 01:28:14.987373 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.987355    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6"} err="failed to get container status \"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6\": rpc error: code = NotFound desc = could not find container \"9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6\": container with ID starting with 9a975a8a34bc67e42f665a505a9682f202b26fd99eab0941e5a6f022c9fce2f6 not found: ID does not exist"
Jul 28 01:28:14.987417 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.987374    2579 scope.go:117] "RemoveContainer" containerID="f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62"
Jul 28 01:28:14.987588 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.987570    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62"} err="failed to get container status \"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62\": rpc error: code = NotFound desc = could not find container \"f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62\": container with ID starting with f754fa099bda2dfa4fba40b077a9d8026e14c664601f6e197428ee221b73cc62 not found: ID does not exist"
Jul 28 01:28:14.987651 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.987590    2579 scope.go:117] "RemoveContainer" containerID="6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c"
Jul 28 01:28:14.987811 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.987790    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c"} err="failed to get container status \"6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c\": rpc error: code = NotFound desc = could not find container \"6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c\": container with ID starting with 6a9b2dbccc6bad66887dc9b6dd3d81c7d3af51fc4975c1e5bd6dec2d95301a0c not found: ID does not exist"
Jul 28 01:28:14.991104 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991081    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:28:14.991384 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991368    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="prom-label-proxy"
Jul 28 01:28:14.991453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991386    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="prom-label-proxy"
Jul 28 01:28:14.991453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991401    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="config-reloader"
Jul 28 01:28:14.991453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991409    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="config-reloader"
Jul 28 01:28:14.991453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991430    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="init-config-reloader"
Jul 28 01:28:14.991453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991439    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="init-config-reloader"
Jul 28 01:28:14.991453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991450    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="alertmanager"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991458    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="alertmanager"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991473    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy-web"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991504    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy-web"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991517    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy-metric"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991525    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy-metric"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991534    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97476fe7-be95-42c8-bbc8-449b21b8d5c8" containerName="console"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991542    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97476fe7-be95-42c8-bbc8-449b21b8d5c8" containerName="console"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991561    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991623    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy-metric"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991635    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="config-reloader"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991644    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97476fe7-be95-42c8-bbc8-449b21b8d5c8" containerName="console"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991653    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="alertmanager"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991661    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy-web"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991669    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="kube-rbac-proxy"
Jul 28 01:28:14.991759 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.991679    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" containerName="prom-label-proxy"
Jul 28 01:28:14.995188 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.995171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:14.998026 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998003    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 01:28:14.998121 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 01:28:14.998121 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998010    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 01:28:14.998121 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998072    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 01:28:14.998121 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998071    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 01:28:14.998121 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998027    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 01:28:14.998424 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 01:28:14.998517 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998472    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-28q42\""
Jul 28 01:28:14.998517 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:14.998477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 01:28:15.003575 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.003557    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 01:28:15.010321 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.010302    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:28:15.091690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03dba6f2-b5b4-4a78-b505-44a084c41ff9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt878\" (UniqueName: \"kubernetes.io/projected/03dba6f2-b5b4-4a78-b505-44a084c41ff9-kube-api-access-gt878\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03dba6f2-b5b4-4a78-b505-44a084c41ff9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03dba6f2-b5b4-4a78-b505-44a084c41ff9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03dba6f2-b5b4-4a78-b505-44a084c41ff9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-web-config\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.091886 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.092115 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03dba6f2-b5b4-4a78-b505-44a084c41ff9-config-out\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.092115 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.091953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-config-volume\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.162534 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.162501    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44145776-6669-4725-8d4b-17aa70cd40ce" path="/var/lib/kubelet/pods/44145776-6669-4725-8d4b-17aa70cd40ce/volumes"
Jul 28 01:28:15.193101 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-config-volume\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193208 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03dba6f2-b5b4-4a78-b505-44a084c41ff9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193208 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193208 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt878\" (UniqueName: \"kubernetes.io/projected/03dba6f2-b5b4-4a78-b505-44a084c41ff9-kube-api-access-gt878\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193208 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03dba6f2-b5b4-4a78-b505-44a084c41ff9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193396 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03dba6f2-b5b4-4a78-b505-44a084c41ff9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193396 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193396 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03dba6f2-b5b4-4a78-b505-44a084c41ff9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193396 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193396 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-web-config\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193396 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193396 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.193723 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.193406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03dba6f2-b5b4-4a78-b505-44a084c41ff9-config-out\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.194303 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.194104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03dba6f2-b5b4-4a78-b505-44a084c41ff9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.194303 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.194163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03dba6f2-b5b4-4a78-b505-44a084c41ff9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.194303 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.194187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/03dba6f2-b5b4-4a78-b505-44a084c41ff9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.196242 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.196217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/03dba6f2-b5b4-4a78-b505-44a084c41ff9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.196343 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.196257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/03dba6f2-b5b4-4a78-b505-44a084c41ff9-config-out\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.196407 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.196361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-web-config\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.196467 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.196401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.196467 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.196413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-config-volume\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.196778 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.196754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.196842 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.196754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.197124 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.197109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.198006 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.197990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/03dba6f2-b5b4-4a78-b505-44a084c41ff9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.200849 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.200828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt878\" (UniqueName: \"kubernetes.io/projected/03dba6f2-b5b4-4a78-b505-44a084c41ff9-kube-api-access-gt878\") pod \"alertmanager-main-0\" (UID: \"03dba6f2-b5b4-4a78-b505-44a084c41ff9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.304319 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.304276    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 01:28:15.439827 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.439795    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 01:28:15.441133 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:28:15.441095    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dba6f2_b5b4_4a78_b505_44a084c41ff9.slice/crio-cabb4b6f284bc6ce26a486d9eed45539a81bc60373ca719d9dc84cf7f0224a90 WatchSource:0}: Error finding container cabb4b6f284bc6ce26a486d9eed45539a81bc60373ca719d9dc84cf7f0224a90: Status 404 returned error can't find the container with id cabb4b6f284bc6ce26a486d9eed45539a81bc60373ca719d9dc84cf7f0224a90
Jul 28 01:28:15.940505 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.940451    2579 generic.go:356] "Generic (PLEG): container finished" podID="03dba6f2-b5b4-4a78-b505-44a084c41ff9" containerID="47350c8a55341a7fb3b534b1dac900599834a94d203c2773ab9f87ed4ee1554b" exitCode=0
Jul 28 01:28:15.940947 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.940544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03dba6f2-b5b4-4a78-b505-44a084c41ff9","Type":"ContainerDied","Data":"47350c8a55341a7fb3b534b1dac900599834a94d203c2773ab9f87ed4ee1554b"}
Jul 28 01:28:15.940947 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:15.940630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03dba6f2-b5b4-4a78-b505-44a084c41ff9","Type":"ContainerStarted","Data":"cabb4b6f284bc6ce26a486d9eed45539a81bc60373ca719d9dc84cf7f0224a90"}
Jul 28 01:28:16.947969 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:16.947934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03dba6f2-b5b4-4a78-b505-44a084c41ff9","Type":"ContainerStarted","Data":"fcbca31db316649b8ae94d4ca80154438d627b0eba81e06f06670ace3750ac9f"}
Jul 28 01:28:16.948439 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:16.948006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03dba6f2-b5b4-4a78-b505-44a084c41ff9","Type":"ContainerStarted","Data":"6aa2cd2377ab3a6fbf7976704c6947295bd2d410894d7697876424eb8c01d99a"}
Jul 28 01:28:16.948439 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:16.948053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03dba6f2-b5b4-4a78-b505-44a084c41ff9","Type":"ContainerStarted","Data":"4a0b7a5c92997fb05973ba5b46e61b069e1ffe7c0edaa94efa94813cd1085fe4"}
Jul 28 01:28:16.948439 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:16.948097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03dba6f2-b5b4-4a78-b505-44a084c41ff9","Type":"ContainerStarted","Data":"b00c835a55088697283ce9f480fe6be9f648f7f7db5cbe1f8cc51183ec0c8ba5"}
Jul 28 01:28:16.948439 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:16.948145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03dba6f2-b5b4-4a78-b505-44a084c41ff9","Type":"ContainerStarted","Data":"87ab5e8c6487da8fcd15abf3bcd77991b0c98c86962c3eda2b01d76eeb05726a"}
Jul 28 01:28:16.948439 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:16.948185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"03dba6f2-b5b4-4a78-b505-44a084c41ff9","Type":"ContainerStarted","Data":"9b2f250d4aa82a072225db25fb867b19a3d2a38f8b059cba419ee2b481272529"}
Jul 28 01:28:16.978010 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:16.977960    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.962184959 podStartE2EDuration="2.977944922s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.759963ms" isStatefulPod=false podCreationTimestamp="2026-07-28 01:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:28:16.975792981 +0000 UTC m=+156.282281761" watchObservedRunningTime="2026-07-28 01:28:16.977944922 +0000 UTC m=+156.284433702"
Jul 28 01:28:40.471272 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.471200    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9fbb6d9f-v7qp5"]
Jul 28 01:28:40.474257 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.474241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.484756 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.484732    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9fbb6d9f-v7qp5"]
Jul 28 01:28:40.596331 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.596291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-service-ca\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.596563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.596344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-trusted-ca-bundle\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.596563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.596364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-oauth-serving-cert\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.596563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.596447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-serving-cert\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.596730 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.596561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-console-config\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.596730 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.596601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-oauth-config\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.596730 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.596626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cntgv\" (UniqueName: \"kubernetes.io/projected/7b23efa1-6306-4879-8d27-2a334daac420-kube-api-access-cntgv\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.697147 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.697112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-service-ca\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.697237 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.697162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-trusted-ca-bundle\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.697303 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.697280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-oauth-serving-cert\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.697377 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.697365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-serving-cert\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.697454 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.697442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-console-config\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.697536 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.697505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-oauth-config\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.697598 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.697542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cntgv\" (UniqueName: \"kubernetes.io/projected/7b23efa1-6306-4879-8d27-2a334daac420-kube-api-access-cntgv\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.697825 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.697802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-service-ca\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.698030 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.698004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-oauth-serving-cert\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.698096 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.698016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-trusted-ca-bundle\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.698140 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.698091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-console-config\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.699762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.699739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-serving-cert\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.699847 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.699830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-oauth-config\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.706008 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.705984    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cntgv\" (UniqueName: \"kubernetes.io/projected/7b23efa1-6306-4879-8d27-2a334daac420-kube-api-access-cntgv\") pod \"console-9fbb6d9f-v7qp5\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") " pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.783657 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.783576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:40.909150 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:40.909127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9fbb6d9f-v7qp5"]
Jul 28 01:28:40.911265 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:28:40.911239    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b23efa1_6306_4879_8d27_2a334daac420.slice/crio-6c239da7c8a79ce22e66b0f0ff169b89fbf4bf21231c13c5406c51d4f2c71987 WatchSource:0}: Error finding container 6c239da7c8a79ce22e66b0f0ff169b89fbf4bf21231c13c5406c51d4f2c71987: Status 404 returned error can't find the container with id 6c239da7c8a79ce22e66b0f0ff169b89fbf4bf21231c13c5406c51d4f2c71987
Jul 28 01:28:41.036025 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:41.035947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fbb6d9f-v7qp5" event={"ID":"7b23efa1-6306-4879-8d27-2a334daac420","Type":"ContainerStarted","Data":"546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22"}
Jul 28 01:28:41.036157 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:41.036041    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fbb6d9f-v7qp5" event={"ID":"7b23efa1-6306-4879-8d27-2a334daac420","Type":"ContainerStarted","Data":"6c239da7c8a79ce22e66b0f0ff169b89fbf4bf21231c13c5406c51d4f2c71987"}
Jul 28 01:28:41.056766 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:41.056719    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9fbb6d9f-v7qp5" podStartSLOduration=1.056703767 podStartE2EDuration="1.056703767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:28:41.055941258 +0000 UTC m=+180.362430058" watchObservedRunningTime="2026-07-28 01:28:41.056703767 +0000 UTC m=+180.363192611"
Jul 28 01:28:50.784373 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:50.784319    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:50.784804 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:50.784425    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:50.789122 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:50.789099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:51.076312 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:51.076237    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:28:51.124741 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:28:51.124711    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c75fbb856-gx8kp"]
Jul 28 01:29:16.145965 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.145907    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c75fbb856-gx8kp" podUID="71055f96-9be9-4dcf-95f9-b16b1abfb65f" containerName="console" containerID="cri-o://d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32" gracePeriod=15
Jul 28 01:29:16.392373 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.392351    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:29:16.478869 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.478797    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqsbf\" (UniqueName: \"kubernetes.io/projected/71055f96-9be9-4dcf-95f9-b16b1abfb65f-kube-api-access-gqsbf\") pod \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") "
Jul 28 01:29:16.478869 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.478835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-service-ca\") pod \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") "
Jul 28 01:29:16.478869 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.478858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-config\") pod \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") "
Jul 28 01:29:16.479102 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.478886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-serving-cert\") pod \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") "
Jul 28 01:29:16.479102 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.478924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-trusted-ca-bundle\") pod \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") "
Jul 28 01:29:16.479102 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.478948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-oauth-serving-cert\") pod \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") "
Jul 28 01:29:16.479102 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.478973    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-oauth-config\") pod \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\" (UID: \"71055f96-9be9-4dcf-95f9-b16b1abfb65f\") "
Jul 28 01:29:16.479306 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.479193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-service-ca" (OuterVolumeSpecName: "service-ca") pod "71055f96-9be9-4dcf-95f9-b16b1abfb65f" (UID: "71055f96-9be9-4dcf-95f9-b16b1abfb65f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:16.479306 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.479241    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-config" (OuterVolumeSpecName: "console-config") pod "71055f96-9be9-4dcf-95f9-b16b1abfb65f" (UID: "71055f96-9be9-4dcf-95f9-b16b1abfb65f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:16.479417 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.479362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "71055f96-9be9-4dcf-95f9-b16b1abfb65f" (UID: "71055f96-9be9-4dcf-95f9-b16b1abfb65f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:16.479417 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.479385    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-service-ca\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:29:16.479417 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.479392    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "71055f96-9be9-4dcf-95f9-b16b1abfb65f" (UID: "71055f96-9be9-4dcf-95f9-b16b1abfb65f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:29:16.479560 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.479407    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:29:16.481044 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.481013    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "71055f96-9be9-4dcf-95f9-b16b1abfb65f" (UID: "71055f96-9be9-4dcf-95f9-b16b1abfb65f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:29:16.481044 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.481039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "71055f96-9be9-4dcf-95f9-b16b1abfb65f" (UID: "71055f96-9be9-4dcf-95f9-b16b1abfb65f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:29:16.481175 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.481066    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71055f96-9be9-4dcf-95f9-b16b1abfb65f-kube-api-access-gqsbf" (OuterVolumeSpecName: "kube-api-access-gqsbf") pod "71055f96-9be9-4dcf-95f9-b16b1abfb65f" (UID: "71055f96-9be9-4dcf-95f9-b16b1abfb65f"). InnerVolumeSpecName "kube-api-access-gqsbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:29:16.580139 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.580113    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqsbf\" (UniqueName: \"kubernetes.io/projected/71055f96-9be9-4dcf-95f9-b16b1abfb65f-kube-api-access-gqsbf\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:29:16.580139 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.580134    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-serving-cert\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:29:16.580139 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.580143    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-trusted-ca-bundle\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:29:16.580316 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.580152    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/71055f96-9be9-4dcf-95f9-b16b1abfb65f-oauth-serving-cert\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:29:16.580316 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:16.580162    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/71055f96-9be9-4dcf-95f9-b16b1abfb65f-console-oauth-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:29:17.164657 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.164620    2579 generic.go:356] "Generic (PLEG): container finished" podID="71055f96-9be9-4dcf-95f9-b16b1abfb65f" containerID="d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32" exitCode=0
Jul 28 01:29:17.164657 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.164660    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c75fbb856-gx8kp"
Jul 28 01:29:17.165154 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.164698    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c75fbb856-gx8kp" event={"ID":"71055f96-9be9-4dcf-95f9-b16b1abfb65f","Type":"ContainerDied","Data":"d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32"}
Jul 28 01:29:17.165154 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.164752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c75fbb856-gx8kp" event={"ID":"71055f96-9be9-4dcf-95f9-b16b1abfb65f","Type":"ContainerDied","Data":"a8429442dacc45fef3cf6ee030d41e3e53886c02db172358288d4a41fd7706f5"}
Jul 28 01:29:17.165154 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.164783    2579 scope.go:117] "RemoveContainer" containerID="d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32"
Jul 28 01:29:17.173836 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.173820    2579 scope.go:117] "RemoveContainer" containerID="d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32"
Jul 28 01:29:17.174082 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:29:17.174061    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32\": container with ID starting with d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32 not found: ID does not exist" containerID="d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32"
Jul 28 01:29:17.174145 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.174089    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32"} err="failed to get container status \"d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32\": rpc error: code = NotFound desc = could not find container \"d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32\": container with ID starting with d0a7d03fcfab0bf132113c6036e0f79e728767480a78f56c736578189c205c32 not found: ID does not exist"
Jul 28 01:29:17.201638 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.201617    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c75fbb856-gx8kp"]
Jul 28 01:29:17.203920 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:17.203901    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c75fbb856-gx8kp"]
Jul 28 01:29:19.163118 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:29:19.163075    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71055f96-9be9-4dcf-95f9-b16b1abfb65f" path="/var/lib/kubelet/pods/71055f96-9be9-4dcf-95f9-b16b1abfb65f/volumes"
Jul 28 01:30:41.039409 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:41.039383    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:30:41.044270 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:41.044250    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:30:41.048006 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:41.047892    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 01:30:56.650870 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.650828    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"]
Jul 28 01:30:56.653155 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.651100    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71055f96-9be9-4dcf-95f9-b16b1abfb65f" containerName="console"
Jul 28 01:30:56.653155 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.651111    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71055f96-9be9-4dcf-95f9-b16b1abfb65f" containerName="console"
Jul 28 01:30:56.653155 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.651165    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71055f96-9be9-4dcf-95f9-b16b1abfb65f" containerName="console"
Jul 28 01:30:56.653997 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.653981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.656552 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.656526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 01:30:56.656763 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.656747    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 01:30:56.657411 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.657393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-fc67x\""
Jul 28 01:30:56.684523 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.684500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvj4\" (UniqueName: \"kubernetes.io/projected/d9b41ecb-f33a-4328-8a3c-a932571ccf62-kube-api-access-zfvj4\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.684620 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.684558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.684620 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.684590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.708769 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.708746    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"]
Jul 28 01:30:56.785684 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.785653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvj4\" (UniqueName: \"kubernetes.io/projected/d9b41ecb-f33a-4328-8a3c-a932571ccf62-kube-api-access-zfvj4\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.785847 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.785722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.785847 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.785754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.786176 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.786155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.786176 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.786166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.793813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.793790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvj4\" (UniqueName: \"kubernetes.io/projected/d9b41ecb-f33a-4328-8a3c-a932571ccf62-kube-api-access-zfvj4\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:56.962903 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:56.962816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:30:57.087174 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:57.087140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"]
Jul 28 01:30:57.090110 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:30:57.090086    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b41ecb_f33a_4328_8a3c_a932571ccf62.slice/crio-9f49a868f3232e8ca5ba5bb762a0276c349df0e4231fc5544b87dc8b8044aec7 WatchSource:0}: Error finding container 9f49a868f3232e8ca5ba5bb762a0276c349df0e4231fc5544b87dc8b8044aec7: Status 404 returned error can't find the container with id 9f49a868f3232e8ca5ba5bb762a0276c349df0e4231fc5544b87dc8b8044aec7
Jul 28 01:30:57.091959 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:57.091939    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:30:57.507157 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:30:57.507118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb" event={"ID":"d9b41ecb-f33a-4328-8a3c-a932571ccf62","Type":"ContainerStarted","Data":"9f49a868f3232e8ca5ba5bb762a0276c349df0e4231fc5544b87dc8b8044aec7"}
Jul 28 01:31:03.531506 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:03.531456    2579 generic.go:356] "Generic (PLEG): container finished" podID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerID="ceb82b13e50663b9936f6f4861a43df7b571da6aaac33f48483e0e59755ce99d" exitCode=0
Jul 28 01:31:03.531864 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:03.531546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb" event={"ID":"d9b41ecb-f33a-4328-8a3c-a932571ccf62","Type":"ContainerDied","Data":"ceb82b13e50663b9936f6f4861a43df7b571da6aaac33f48483e0e59755ce99d"}
Jul 28 01:31:05.541835 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:05.541800    2579 generic.go:356] "Generic (PLEG): container finished" podID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerID="66f0cbb318059ddba6812c9893a6f5e596d9313b9e1b34a3e5ad82ac3e61267b" exitCode=0
Jul 28 01:31:05.542231 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:05.541880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb" event={"ID":"d9b41ecb-f33a-4328-8a3c-a932571ccf62","Type":"ContainerDied","Data":"66f0cbb318059ddba6812c9893a6f5e596d9313b9e1b34a3e5ad82ac3e61267b"}
Jul 28 01:31:13.576287 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:13.576253    2579 generic.go:356] "Generic (PLEG): container finished" podID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerID="56062f788ed9e772b6626ec1d7a57f2c7533a486bf5c4df6f9bf2609eab63ee5" exitCode=0
Jul 28 01:31:13.576713 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:13.576335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb" event={"ID":"d9b41ecb-f33a-4328-8a3c-a932571ccf62","Type":"ContainerDied","Data":"56062f788ed9e772b6626ec1d7a57f2c7533a486bf5c4df6f9bf2609eab63ee5"}
Jul 28 01:31:14.700517 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.700477    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:31:14.839534 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.839431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-util\") pod \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") "
Jul 28 01:31:14.839534 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.839520    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfvj4\" (UniqueName: \"kubernetes.io/projected/d9b41ecb-f33a-4328-8a3c-a932571ccf62-kube-api-access-zfvj4\") pod \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") "
Jul 28 01:31:14.839764 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.839541    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-bundle\") pod \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\" (UID: \"d9b41ecb-f33a-4328-8a3c-a932571ccf62\") "
Jul 28 01:31:14.840187 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.840159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-bundle" (OuterVolumeSpecName: "bundle") pod "d9b41ecb-f33a-4328-8a3c-a932571ccf62" (UID: "d9b41ecb-f33a-4328-8a3c-a932571ccf62"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:31:14.841689 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.841667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b41ecb-f33a-4328-8a3c-a932571ccf62-kube-api-access-zfvj4" (OuterVolumeSpecName: "kube-api-access-zfvj4") pod "d9b41ecb-f33a-4328-8a3c-a932571ccf62" (UID: "d9b41ecb-f33a-4328-8a3c-a932571ccf62"). InnerVolumeSpecName "kube-api-access-zfvj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:31:14.843692 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.843670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-util" (OuterVolumeSpecName: "util") pod "d9b41ecb-f33a-4328-8a3c-a932571ccf62" (UID: "d9b41ecb-f33a-4328-8a3c-a932571ccf62"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:31:14.940252 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.940214    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-util\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:31:14.940252 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.940245    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvj4\" (UniqueName: \"kubernetes.io/projected/d9b41ecb-f33a-4328-8a3c-a932571ccf62-kube-api-access-zfvj4\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:31:14.940252 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:14.940257    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9b41ecb-f33a-4328-8a3c-a932571ccf62-bundle\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:31:15.586215 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:15.586139    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb"
Jul 28 01:31:15.586356 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:15.586138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5qbmfb" event={"ID":"d9b41ecb-f33a-4328-8a3c-a932571ccf62","Type":"ContainerDied","Data":"9f49a868f3232e8ca5ba5bb762a0276c349df0e4231fc5544b87dc8b8044aec7"}
Jul 28 01:31:15.586356 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:15.586258    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f49a868f3232e8ca5ba5bb762a0276c349df0e4231fc5544b87dc8b8044aec7"
Jul 28 01:31:19.593260 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.593172    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"]
Jul 28 01:31:19.593702 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.593583    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerName="util"
Jul 28 01:31:19.593702 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.593600    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerName="util"
Jul 28 01:31:19.593702 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.593625    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerName="pull"
Jul 28 01:31:19.593702 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.593632    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerName="pull"
Jul 28 01:31:19.593702 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.593656    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerName="extract"
Jul 28 01:31:19.593702 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.593664    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerName="extract"
Jul 28 01:31:19.593994 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.593724    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9b41ecb-f33a-4328-8a3c-a932571ccf62" containerName="extract"
Jul 28 01:31:19.637197 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.637168    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"]
Jul 28 01:31:19.637349 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.637273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:19.639742 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.639715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 01:31:19.639879 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.639853    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-4cz67\""
Jul 28 01:31:19.639879 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.639861    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 01:31:19.640010 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.639900    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 01:31:19.776014 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.775975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfqq4\" (UniqueName: \"kubernetes.io/projected/8c30ef49-f3ca-4e8b-9dc8-a581859da39f-kube-api-access-cfqq4\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm\" (UID: \"8c30ef49-f3ca-4e8b-9dc8-a581859da39f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:19.776198 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.776026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8c30ef49-f3ca-4e8b-9dc8-a581859da39f-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm\" (UID: \"8c30ef49-f3ca-4e8b-9dc8-a581859da39f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:19.877356 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.877328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfqq4\" (UniqueName: \"kubernetes.io/projected/8c30ef49-f3ca-4e8b-9dc8-a581859da39f-kube-api-access-cfqq4\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm\" (UID: \"8c30ef49-f3ca-4e8b-9dc8-a581859da39f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:19.877514 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.877372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8c30ef49-f3ca-4e8b-9dc8-a581859da39f-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm\" (UID: \"8c30ef49-f3ca-4e8b-9dc8-a581859da39f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:19.879596 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.879571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8c30ef49-f3ca-4e8b-9dc8-a581859da39f-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm\" (UID: \"8c30ef49-f3ca-4e8b-9dc8-a581859da39f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:19.885729 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.885701    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfqq4\" (UniqueName: \"kubernetes.io/projected/8c30ef49-f3ca-4e8b-9dc8-a581859da39f-kube-api-access-cfqq4\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm\" (UID: \"8c30ef49-f3ca-4e8b-9dc8-a581859da39f\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:19.947246 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:19.947212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:20.086371 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:20.086341    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"]
Jul 28 01:31:20.089802 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:31:20.089778    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c30ef49_f3ca_4e8b_9dc8_a581859da39f.slice/crio-0fd986b51e7afaf7dd7df72375b30df5bad312608a84afda55e78cf029d52aa6 WatchSource:0}: Error finding container 0fd986b51e7afaf7dd7df72375b30df5bad312608a84afda55e78cf029d52aa6: Status 404 returned error can't find the container with id 0fd986b51e7afaf7dd7df72375b30df5bad312608a84afda55e78cf029d52aa6
Jul 28 01:31:20.604589 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:20.604557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm" event={"ID":"8c30ef49-f3ca-4e8b-9dc8-a581859da39f","Type":"ContainerStarted","Data":"0fd986b51e7afaf7dd7df72375b30df5bad312608a84afda55e78cf029d52aa6"}
Jul 28 01:31:24.620631 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:24.620600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm" event={"ID":"8c30ef49-f3ca-4e8b-9dc8-a581859da39f","Type":"ContainerStarted","Data":"13f4ceafc6ab86a528734d2ce2a6615e7b07b9a5a9193ea7e5079ad04e050505"}
Jul 28 01:31:24.620980 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:24.620738    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:24.648006 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:24.647956    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm" podStartSLOduration=1.6091703769999999 podStartE2EDuration="5.647941502s" totalImagesPullingTime="4.038771125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:31:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:31:24.64522884 +0000 UTC m=+343.951717621" watchObservedRunningTime="2026-07-28 01:31:24.647941502 +0000 UTC m=+343.954430283"
Jul 28 01:31:25.023305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.023230    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-8xv8m"]
Jul 28 01:31:25.038353 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.038320    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-8xv8m"]
Jul 28 01:31:25.038511 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.038420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.040700 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.040677    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 01:31:25.040815 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.040764    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-rrx2z\""
Jul 28 01:31:25.040815 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.040787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 01:31:25.120176 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.120143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-certificates\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.120176 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.120179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c49865cc-eb85-46e6-887a-83879a8ca029-cabundle0\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.120354 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.120218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwgst\" (UniqueName: \"kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-kube-api-access-fwgst\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.221743 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.221158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwgst\" (UniqueName: \"kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-kube-api-access-fwgst\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.221743 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.221277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-certificates\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.221743 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.221302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c49865cc-eb85-46e6-887a-83879a8ca029-cabundle0\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.221743 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:31:25.221493    2579 secret.go:281] references non-existent secret key: ca.crt
Jul 28 01:31:25.221743 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:31:25.221514    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 01:31:25.221743 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:31:25.221524    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-8xv8m: references non-existent secret key: ca.crt
Jul 28 01:31:25.221743 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:31:25.221583    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-certificates podName:c49865cc-eb85-46e6-887a-83879a8ca029 nodeName:}" failed. No retries permitted until 2026-07-28 01:31:25.721561813 +0000 UTC m=+345.028050575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-certificates") pod "keda-operator-77f648d694-8xv8m" (UID: "c49865cc-eb85-46e6-887a-83879a8ca029") : references non-existent secret key: ca.crt
Jul 28 01:31:25.222213 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.222125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c49865cc-eb85-46e6-887a-83879a8ca029-cabundle0\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.230054 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.230028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwgst\" (UniqueName: \"kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-kube-api-access-fwgst\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.451784 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.451748    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"]
Jul 28 01:31:25.453900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.453884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.457709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.457684    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 01:31:25.468114 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.468093    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"]
Jul 28 01:31:25.523625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.523599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/22070ee5-32f0-4d07-b882-9617efc95277-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.523784 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.523638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/22070ee5-32f0-4d07-b882-9617efc95277-certificates\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.523784 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.523716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ppms\" (UniqueName: \"kubernetes.io/projected/22070ee5-32f0-4d07-b882-9617efc95277-kube-api-access-2ppms\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.624394 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.624370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ppms\" (UniqueName: \"kubernetes.io/projected/22070ee5-32f0-4d07-b882-9617efc95277-kube-api-access-2ppms\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.624824 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.624404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/22070ee5-32f0-4d07-b882-9617efc95277-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.624824 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.624514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/22070ee5-32f0-4d07-b882-9617efc95277-certificates\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.624936 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.624921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/22070ee5-32f0-4d07-b882-9617efc95277-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.627071 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.627049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/22070ee5-32f0-4d07-b882-9617efc95277-certificates\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.635502 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.635457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ppms\" (UniqueName: \"kubernetes.io/projected/22070ee5-32f0-4d07-b882-9617efc95277-kube-api-access-2ppms\") pod \"keda-metrics-apiserver-55557df8c-gfqpc\" (UID: \"22070ee5-32f0-4d07-b882-9617efc95277\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.725956 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.725867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-certificates\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.728385 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.728357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c49865cc-eb85-46e6-887a-83879a8ca029-certificates\") pod \"keda-operator-77f648d694-8xv8m\" (UID: \"c49865cc-eb85-46e6-887a-83879a8ca029\") " pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:25.764266 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.764238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:25.806618 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.806586    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-n7jq5"]
Jul 28 01:31:25.809098 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.809075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:25.811861 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.811664    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 28 01:31:25.826949 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.826872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-n7jq5"]
Jul 28 01:31:25.899270 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.899245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"]
Jul 28 01:31:25.901495 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:31:25.901448    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22070ee5_32f0_4d07_b882_9617efc95277.slice/crio-8d72425e303aef9015a0e4e9df919853051ed2f18b8102707f53ee693aa00ade WatchSource:0}: Error finding container 8d72425e303aef9015a0e4e9df919853051ed2f18b8102707f53ee693aa00ade: Status 404 returned error can't find the container with id 8d72425e303aef9015a0e4e9df919853051ed2f18b8102707f53ee693aa00ade
Jul 28 01:31:25.928374 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.928340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48sb7\" (UniqueName: \"kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-kube-api-access-48sb7\") pod \"keda-admission-56f9798bb4-n7jq5\" (UID: \"1758a668-1f65-48ae-a8d7-710abb522f52\") " pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:25.928525 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.928383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-certificates\") pod \"keda-admission-56f9798bb4-n7jq5\" (UID: \"1758a668-1f65-48ae-a8d7-710abb522f52\") " pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:25.949319 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:25.949291    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:26.029142 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.029100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48sb7\" (UniqueName: \"kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-kube-api-access-48sb7\") pod \"keda-admission-56f9798bb4-n7jq5\" (UID: \"1758a668-1f65-48ae-a8d7-710abb522f52\") " pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:26.029296 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.029165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-certificates\") pod \"keda-admission-56f9798bb4-n7jq5\" (UID: \"1758a668-1f65-48ae-a8d7-710abb522f52\") " pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:26.029364 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:31:26.029336    2579 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 28 01:31:26.029364 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:31:26.029359    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-n7jq5: secret "keda-admission-webhooks-certs" not found
Jul 28 01:31:26.029514 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:31:26.029422    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-certificates podName:1758a668-1f65-48ae-a8d7-710abb522f52 nodeName:}" failed. No retries permitted until 2026-07-28 01:31:26.52939952 +0000 UTC m=+345.835888298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-certificates") pod "keda-admission-56f9798bb4-n7jq5" (UID: "1758a668-1f65-48ae-a8d7-710abb522f52") : secret "keda-admission-webhooks-certs" not found
Jul 28 01:31:26.037779 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.037753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48sb7\" (UniqueName: \"kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-kube-api-access-48sb7\") pod \"keda-admission-56f9798bb4-n7jq5\" (UID: \"1758a668-1f65-48ae-a8d7-710abb522f52\") " pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:26.074345 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.074316    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-8xv8m"]
Jul 28 01:31:26.076659 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:31:26.076633    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49865cc_eb85_46e6_887a_83879a8ca029.slice/crio-19c4cbadf37bb129bc91a1510ea26f75bd72d5d399e5dccfc5a74de66960af62 WatchSource:0}: Error finding container 19c4cbadf37bb129bc91a1510ea26f75bd72d5d399e5dccfc5a74de66960af62: Status 404 returned error can't find the container with id 19c4cbadf37bb129bc91a1510ea26f75bd72d5d399e5dccfc5a74de66960af62
Jul 28 01:31:26.535398 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.535361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-certificates\") pod \"keda-admission-56f9798bb4-n7jq5\" (UID: \"1758a668-1f65-48ae-a8d7-710abb522f52\") " pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:26.538733 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.538690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1758a668-1f65-48ae-a8d7-710abb522f52-certificates\") pod \"keda-admission-56f9798bb4-n7jq5\" (UID: \"1758a668-1f65-48ae-a8d7-710abb522f52\") " pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:26.629694 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.629657    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc" event={"ID":"22070ee5-32f0-4d07-b882-9617efc95277","Type":"ContainerStarted","Data":"8d72425e303aef9015a0e4e9df919853051ed2f18b8102707f53ee693aa00ade"}
Jul 28 01:31:26.631034 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.631002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-8xv8m" event={"ID":"c49865cc-eb85-46e6-887a-83879a8ca029","Type":"ContainerStarted","Data":"19c4cbadf37bb129bc91a1510ea26f75bd72d5d399e5dccfc5a74de66960af62"}
Jul 28 01:31:26.721323 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.721290    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:26.908556 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:26.908430    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-n7jq5"]
Jul 28 01:31:26.911140 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:31:26.911061    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1758a668_1f65_48ae_a8d7_710abb522f52.slice/crio-d773e8d34ec12caa7ee384f00ae41083b6dc87a5f612e5f96c948ec8bc95c012 WatchSource:0}: Error finding container d773e8d34ec12caa7ee384f00ae41083b6dc87a5f612e5f96c948ec8bc95c012: Status 404 returned error can't find the container with id d773e8d34ec12caa7ee384f00ae41083b6dc87a5f612e5f96c948ec8bc95c012
Jul 28 01:31:27.637293 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:27.637242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-n7jq5" event={"ID":"1758a668-1f65-48ae-a8d7-710abb522f52","Type":"ContainerStarted","Data":"d773e8d34ec12caa7ee384f00ae41083b6dc87a5f612e5f96c948ec8bc95c012"}
Jul 28 01:31:28.645233 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:28.645136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc" event={"ID":"22070ee5-32f0-4d07-b882-9617efc95277","Type":"ContainerStarted","Data":"3251bf5000419cba082d25906a9d81b738e0e0b2967aa48dfda04c9e66693fcc"}
Jul 28 01:31:28.645705 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:28.645246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:29.652982 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:29.652950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-n7jq5" event={"ID":"1758a668-1f65-48ae-a8d7-710abb522f52","Type":"ContainerStarted","Data":"8c397fc2486b3dbf5e403f8c659c4ebc43b51ebc77813dae1e6cf91df73796da"}
Jul 28 01:31:29.669498 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:29.669417    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-n7jq5" podStartSLOduration=2.472276472 podStartE2EDuration="4.6694024s" totalImagesPullingTime="2.197125928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:31:29.668186573 +0000 UTC m=+348.974675356" watchObservedRunningTime="2026-07-28 01:31:29.6694024 +0000 UTC m=+348.975891181"
Jul 28 01:31:29.669931 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:29.669897    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc" podStartSLOduration=2.202974307 podStartE2EDuration="4.669887895s" totalImagesPullingTime="2.466913588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:31:28.684162387 +0000 UTC m=+347.990651167" watchObservedRunningTime="2026-07-28 01:31:29.669887895 +0000 UTC m=+348.976376719"
Jul 28 01:31:30.658681 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:30.658644    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:31.663352 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:31.663316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-8xv8m" event={"ID":"c49865cc-eb85-46e6-887a-83879a8ca029","Type":"ContainerStarted","Data":"b68402579269b7f3c623c0eb5c185dd6e701ea3aa1b56d1846e9d3e5bc43c4d1"}
Jul 28 01:31:31.663741 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:31.663508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:31:31.686952 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:31.686911    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-8xv8m" podStartSLOduration=1.8078387679999999 podStartE2EDuration="6.686896203s" totalImagesPullingTime="4.879057435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:31:31.686032383 +0000 UTC m=+350.992521165" watchObservedRunningTime="2026-07-28 01:31:31.686896203 +0000 UTC m=+350.993384985"
Jul 28 01:31:39.657442 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:39.657372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-gfqpc"
Jul 28 01:31:45.627169 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:45.627140    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-nbfvm"
Jul 28 01:31:51.666003 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:51.665973    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-n7jq5"
Jul 28 01:31:52.670387 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:31:52.670354    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-8xv8m"
Jul 28 01:32:41.298444 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.298406    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dd55b56f5-zp9wm"]
Jul 28 01:32:41.300823 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.300801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.320546 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.320521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dd55b56f5-zp9wm"]
Jul 28 01:32:41.342812 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.342777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-trusted-ca-bundle\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.342957 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.342817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be39a079-a272-4b93-be94-64e19d822653-console-oauth-config\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.342957 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.342839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-oauth-serving-cert\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.342957 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.342855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6t6\" (UniqueName: \"kubernetes.io/projected/be39a079-a272-4b93-be94-64e19d822653-kube-api-access-jw6t6\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.342957 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.342891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-service-ca\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.342957 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.342924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be39a079-a272-4b93-be94-64e19d822653-console-serving-cert\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.342957 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.342944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-console-config\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.444231 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.444194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-service-ca\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.444419 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.444250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be39a079-a272-4b93-be94-64e19d822653-console-serving-cert\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.444419 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.444278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-console-config\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.444419 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.444392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-trusted-ca-bundle\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.444602 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.444434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be39a079-a272-4b93-be94-64e19d822653-console-oauth-config\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.444602 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.444464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-oauth-serving-cert\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.444701 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.444624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6t6\" (UniqueName: \"kubernetes.io/projected/be39a079-a272-4b93-be94-64e19d822653-kube-api-access-jw6t6\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.445123 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.445094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-console-config\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.445221 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.445094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-service-ca\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.445265 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.445242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-oauth-serving-cert\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.445265 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.445250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be39a079-a272-4b93-be94-64e19d822653-trusted-ca-bundle\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.447319 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.447295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be39a079-a272-4b93-be94-64e19d822653-console-oauth-config\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.447391 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.447376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be39a079-a272-4b93-be94-64e19d822653-console-serving-cert\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.457039 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.457014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6t6\" (UniqueName: \"kubernetes.io/projected/be39a079-a272-4b93-be94-64e19d822653-kube-api-access-jw6t6\") pod \"console-6dd55b56f5-zp9wm\" (UID: \"be39a079-a272-4b93-be94-64e19d822653\") " pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.609558 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.609528    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:41.747844 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.747812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dd55b56f5-zp9wm"]
Jul 28 01:32:41.751100 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:32:41.751074    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe39a079_a272_4b93_be94_64e19d822653.slice/crio-f3f9dbb32d87473b39a85f9321db5f5f729a1b444065ae0a712c24412dfbe883 WatchSource:0}: Error finding container f3f9dbb32d87473b39a85f9321db5f5f729a1b444065ae0a712c24412dfbe883: Status 404 returned error can't find the container with id f3f9dbb32d87473b39a85f9321db5f5f729a1b444065ae0a712c24412dfbe883
Jul 28 01:32:41.946191 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.946108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dd55b56f5-zp9wm" event={"ID":"be39a079-a272-4b93-be94-64e19d822653","Type":"ContainerStarted","Data":"4b74d3cdff58fac8b5f18658bc98fd4095fd3da77ea52fbe7134ce524c8161d2"}
Jul 28 01:32:41.946191 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.946159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dd55b56f5-zp9wm" event={"ID":"be39a079-a272-4b93-be94-64e19d822653","Type":"ContainerStarted","Data":"f3f9dbb32d87473b39a85f9321db5f5f729a1b444065ae0a712c24412dfbe883"}
Jul 28 01:32:41.976896 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:41.976839    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dd55b56f5-zp9wm" podStartSLOduration=0.976822865 podStartE2EDuration="976.822865ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:32:41.97663913 +0000 UTC m=+421.283127910" watchObservedRunningTime="2026-07-28 01:32:41.976822865 +0000 UTC m=+421.283311647"
Jul 28 01:32:44.032318 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.032275    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-86vg7"]
Jul 28 01:32:44.034942 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.034918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.037846 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.037825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 01:32:44.037974 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.037955    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 01:32:44.038743 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.038723    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 01:32:44.038823 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.038731    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-899f4\""
Jul 28 01:32:44.042680 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.042656    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-wlfjl"]
Jul 28 01:32:44.044724 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.044702    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:44.052373 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.047927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-86vg7"]
Jul 28 01:32:44.052373 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.048780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 01:32:44.052373 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.049035    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-r6f2r\""
Jul 28 01:32:44.061817 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.061643    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-wlfjl"]
Jul 28 01:32:44.066283 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.066260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlz9l\" (UniqueName: \"kubernetes.io/projected/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-kube-api-access-hlz9l\") pod \"kserve-controller-manager-ff96d5959-86vg7\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") " pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.066387 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.066303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz87j\" (UniqueName: \"kubernetes.io/projected/5e777faa-1be8-4281-b7c5-c0839a28b761-kube-api-access-fz87j\") pod \"llmisvc-controller-manager-84fc959947-wlfjl\" (UID: \"5e777faa-1be8-4281-b7c5-c0839a28b761\") " pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:44.066387 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.066336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e777faa-1be8-4281-b7c5-c0839a28b761-cert\") pod \"llmisvc-controller-manager-84fc959947-wlfjl\" (UID: \"5e777faa-1be8-4281-b7c5-c0839a28b761\") " pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:44.066387 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.066373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert\") pod \"kserve-controller-manager-ff96d5959-86vg7\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") " pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.092220 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.092189    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-zm4ml"]
Jul 28 01:32:44.094329 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.094311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:44.096520 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.096496    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 01:32:44.096919 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.096904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9497s\""
Jul 28 01:32:44.106745 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.106722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zm4ml"]
Jul 28 01:32:44.166873 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.166835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz87j\" (UniqueName: \"kubernetes.io/projected/5e777faa-1be8-4281-b7c5-c0839a28b761-kube-api-access-fz87j\") pod \"llmisvc-controller-manager-84fc959947-wlfjl\" (UID: \"5e777faa-1be8-4281-b7c5-c0839a28b761\") " pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:44.167076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.166885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e777faa-1be8-4281-b7c5-c0839a28b761-cert\") pod \"llmisvc-controller-manager-84fc959947-wlfjl\" (UID: \"5e777faa-1be8-4281-b7c5-c0839a28b761\") " pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:44.167076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.166915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert\") pod \"kserve-controller-manager-ff96d5959-86vg7\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") " pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.167076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.166960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjn2h\" (UniqueName: \"kubernetes.io/projected/17899d4e-0142-4ca3-ad08-8ea958f54038-kube-api-access-tjn2h\") pod \"seaweedfs-64568bcd49-zm4ml\" (UID: \"17899d4e-0142-4ca3-ad08-8ea958f54038\") " pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:44.167076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.166995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlz9l\" (UniqueName: \"kubernetes.io/projected/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-kube-api-access-hlz9l\") pod \"kserve-controller-manager-ff96d5959-86vg7\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") " pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.167076 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.167032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/17899d4e-0142-4ca3-ad08-8ea958f54038-data\") pod \"seaweedfs-64568bcd49-zm4ml\" (UID: \"17899d4e-0142-4ca3-ad08-8ea958f54038\") " pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:44.167275 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:32:44.167177    2579 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 01:32:44.167275 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:32:44.167255    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert podName:4677238c-6df9-4ec2-9af2-8c1aad40e7c4 nodeName:}" failed. No retries permitted until 2026-07-28 01:32:44.667233111 +0000 UTC m=+423.973721884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert") pod "kserve-controller-manager-ff96d5959-86vg7" (UID: "4677238c-6df9-4ec2-9af2-8c1aad40e7c4") : secret "kserve-webhook-server-cert" not found
Jul 28 01:32:44.169537 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.169510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e777faa-1be8-4281-b7c5-c0839a28b761-cert\") pod \"llmisvc-controller-manager-84fc959947-wlfjl\" (UID: \"5e777faa-1be8-4281-b7c5-c0839a28b761\") " pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:44.176559 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.176533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz87j\" (UniqueName: \"kubernetes.io/projected/5e777faa-1be8-4281-b7c5-c0839a28b761-kube-api-access-fz87j\") pod \"llmisvc-controller-manager-84fc959947-wlfjl\" (UID: \"5e777faa-1be8-4281-b7c5-c0839a28b761\") " pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:44.177052 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.177033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlz9l\" (UniqueName: \"kubernetes.io/projected/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-kube-api-access-hlz9l\") pod \"kserve-controller-manager-ff96d5959-86vg7\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") " pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.268162 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.268126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/17899d4e-0142-4ca3-ad08-8ea958f54038-data\") pod \"seaweedfs-64568bcd49-zm4ml\" (UID: \"17899d4e-0142-4ca3-ad08-8ea958f54038\") " pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:44.268445 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.268405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn2h\" (UniqueName: \"kubernetes.io/projected/17899d4e-0142-4ca3-ad08-8ea958f54038-kube-api-access-tjn2h\") pod \"seaweedfs-64568bcd49-zm4ml\" (UID: \"17899d4e-0142-4ca3-ad08-8ea958f54038\") " pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:44.268445 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.268432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/17899d4e-0142-4ca3-ad08-8ea958f54038-data\") pod \"seaweedfs-64568bcd49-zm4ml\" (UID: \"17899d4e-0142-4ca3-ad08-8ea958f54038\") " pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:44.287590 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.287528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn2h\" (UniqueName: \"kubernetes.io/projected/17899d4e-0142-4ca3-ad08-8ea958f54038-kube-api-access-tjn2h\") pod \"seaweedfs-64568bcd49-zm4ml\" (UID: \"17899d4e-0142-4ca3-ad08-8ea958f54038\") " pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:44.365232 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.365190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:44.403603 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.403572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:44.503301 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.503269    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-wlfjl"]
Jul 28 01:32:44.504809 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:32:44.504786    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5e777faa_1be8_4281_b7c5_c0839a28b761.slice/crio-03831748963fd703930d8c55d66168cc29e93e825ec3442db046d3df74535323 WatchSource:0}: Error finding container 03831748963fd703930d8c55d66168cc29e93e825ec3442db046d3df74535323: Status 404 returned error can't find the container with id 03831748963fd703930d8c55d66168cc29e93e825ec3442db046d3df74535323
Jul 28 01:32:44.546856 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.546831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-zm4ml"]
Jul 28 01:32:44.548609 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:32:44.548586    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17899d4e_0142_4ca3_ad08_8ea958f54038.slice/crio-f5744b550860425972c7b59b0b595d5518ffdd979a7cc0ca2c1247b358a50a7b WatchSource:0}: Error finding container f5744b550860425972c7b59b0b595d5518ffdd979a7cc0ca2c1247b358a50a7b: Status 404 returned error can't find the container with id f5744b550860425972c7b59b0b595d5518ffdd979a7cc0ca2c1247b358a50a7b
Jul 28 01:32:44.671786 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.671753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert\") pod \"kserve-controller-manager-ff96d5959-86vg7\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") " pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.674035 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.674017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert\") pod \"kserve-controller-manager-ff96d5959-86vg7\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") " pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.947125 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.947089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:44.965368 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.965333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl" event={"ID":"5e777faa-1be8-4281-b7c5-c0839a28b761","Type":"ContainerStarted","Data":"03831748963fd703930d8c55d66168cc29e93e825ec3442db046d3df74535323"}
Jul 28 01:32:44.967207 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:44.967164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zm4ml" event={"ID":"17899d4e-0142-4ca3-ad08-8ea958f54038","Type":"ContainerStarted","Data":"f5744b550860425972c7b59b0b595d5518ffdd979a7cc0ca2c1247b358a50a7b"}
Jul 28 01:32:45.099728 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:45.099706    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-86vg7"]
Jul 28 01:32:45.105078 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:32:45.105048    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4677238c_6df9_4ec2_9af2_8c1aad40e7c4.slice/crio-0a6f616b6d5a1bce1c254258d9c482096bca97fabc4fb70050ea81f30dace190 WatchSource:0}: Error finding container 0a6f616b6d5a1bce1c254258d9c482096bca97fabc4fb70050ea81f30dace190: Status 404 returned error can't find the container with id 0a6f616b6d5a1bce1c254258d9c482096bca97fabc4fb70050ea81f30dace190
Jul 28 01:32:45.978517 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:45.978457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-86vg7" event={"ID":"4677238c-6df9-4ec2-9af2-8c1aad40e7c4","Type":"ContainerStarted","Data":"0a6f616b6d5a1bce1c254258d9c482096bca97fabc4fb70050ea81f30dace190"}
Jul 28 01:32:50.000703 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.000662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-86vg7" event={"ID":"4677238c-6df9-4ec2-9af2-8c1aad40e7c4","Type":"ContainerStarted","Data":"fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f"}
Jul 28 01:32:50.001155 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.000837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:32:50.002260 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.002234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-zm4ml" event={"ID":"17899d4e-0142-4ca3-ad08-8ea958f54038","Type":"ContainerStarted","Data":"e01ddb4aa2162eb7bfa31c7076290f55eb426af892cf495979c299740988b4fb"}
Jul 28 01:32:50.002441 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.002306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:32:50.003791 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.003770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl" event={"ID":"5e777faa-1be8-4281-b7c5-c0839a28b761","Type":"ContainerStarted","Data":"19d4c04f8203195765db6895343b08d795d5705b3bf4a89ae2ae5caba304b30c"}
Jul 28 01:32:50.003889 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.003819    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:32:50.041860 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.041809    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-86vg7" podStartSLOduration=1.954320934 podStartE2EDuration="6.041795994s" totalImagesPullingTime="4.08747506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:32:50.040229594 +0000 UTC m=+429.346718372" watchObservedRunningTime="2026-07-28 01:32:50.041795994 +0000 UTC m=+429.348284774"
Jul 28 01:32:50.061439 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.061393    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-zm4ml" podStartSLOduration=1.361538132 podStartE2EDuration="6.061379557s" totalImagesPullingTime="4.699841425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:32:50.061353301 +0000 UTC m=+429.367842094" watchObservedRunningTime="2026-07-28 01:32:50.061379557 +0000 UTC m=+429.367868339"
Jul 28 01:32:50.078209 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:50.078170    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl" podStartSLOduration=1.388325529 podStartE2EDuration="6.07815735s" totalImagesPullingTime="4.689831821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:32:50.077681442 +0000 UTC m=+429.384170222" watchObservedRunningTime="2026-07-28 01:32:50.07815735 +0000 UTC m=+429.384646130"
Jul 28 01:32:51.610303 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:51.610269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:51.610880 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:51.610338    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:51.614879 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:51.614858    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:52.015363 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:52.015291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dd55b56f5-zp9wm"
Jul 28 01:32:52.062281 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:52.062247    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9fbb6d9f-v7qp5"]
Jul 28 01:32:56.010668 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:32:56.010639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-zm4ml"
Jul 28 01:33:17.081518 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.081417    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9fbb6d9f-v7qp5" podUID="7b23efa1-6306-4879-8d27-2a334daac420" containerName="console" containerID="cri-o://546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22" gracePeriod=15
Jul 28 01:33:17.320896 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.320873    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:33:17.462734 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.462698    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-oauth-serving-cert\") pod \"7b23efa1-6306-4879-8d27-2a334daac420\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") "
Jul 28 01:33:17.462734 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.462738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-service-ca\") pod \"7b23efa1-6306-4879-8d27-2a334daac420\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") "
Jul 28 01:33:17.462963 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.462756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-console-config\") pod \"7b23efa1-6306-4879-8d27-2a334daac420\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") "
Jul 28 01:33:17.462963 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.462804    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-serving-cert\") pod \"7b23efa1-6306-4879-8d27-2a334daac420\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") "
Jul 28 01:33:17.462963 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.462830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-oauth-config\") pod \"7b23efa1-6306-4879-8d27-2a334daac420\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") "
Jul 28 01:33:17.462963 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.462869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-trusted-ca-bundle\") pod \"7b23efa1-6306-4879-8d27-2a334daac420\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") "
Jul 28 01:33:17.462963 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.462905    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cntgv\" (UniqueName: \"kubernetes.io/projected/7b23efa1-6306-4879-8d27-2a334daac420-kube-api-access-cntgv\") pod \"7b23efa1-6306-4879-8d27-2a334daac420\" (UID: \"7b23efa1-6306-4879-8d27-2a334daac420\") "
Jul 28 01:33:17.463335 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.463307    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7b23efa1-6306-4879-8d27-2a334daac420" (UID: "7b23efa1-6306-4879-8d27-2a334daac420"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:33:17.463437 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.463341    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-console-config" (OuterVolumeSpecName: "console-config") pod "7b23efa1-6306-4879-8d27-2a334daac420" (UID: "7b23efa1-6306-4879-8d27-2a334daac420"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:33:17.463437 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.463342    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7b23efa1-6306-4879-8d27-2a334daac420" (UID: "7b23efa1-6306-4879-8d27-2a334daac420"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:33:17.463437 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.463406    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-service-ca" (OuterVolumeSpecName: "service-ca") pod "7b23efa1-6306-4879-8d27-2a334daac420" (UID: "7b23efa1-6306-4879-8d27-2a334daac420"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:33:17.465053 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.465026    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7b23efa1-6306-4879-8d27-2a334daac420" (UID: "7b23efa1-6306-4879-8d27-2a334daac420"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:33:17.465151 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.465081    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b23efa1-6306-4879-8d27-2a334daac420-kube-api-access-cntgv" (OuterVolumeSpecName: "kube-api-access-cntgv") pod "7b23efa1-6306-4879-8d27-2a334daac420" (UID: "7b23efa1-6306-4879-8d27-2a334daac420"). InnerVolumeSpecName "kube-api-access-cntgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:33:17.465151 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.465082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7b23efa1-6306-4879-8d27-2a334daac420" (UID: "7b23efa1-6306-4879-8d27-2a334daac420"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:33:17.564332 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.564285    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-service-ca\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:17.564332 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.564330    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-console-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:17.564332 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.564339    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-serving-cert\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:17.564576 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.564350    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7b23efa1-6306-4879-8d27-2a334daac420-console-oauth-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:17.564576 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.564363    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-trusted-ca-bundle\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:17.564576 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.564372    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cntgv\" (UniqueName: \"kubernetes.io/projected/7b23efa1-6306-4879-8d27-2a334daac420-kube-api-access-cntgv\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:17.564576 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:17.564382    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7b23efa1-6306-4879-8d27-2a334daac420-oauth-serving-cert\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:18.124642 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.124607    2579 generic.go:356] "Generic (PLEG): container finished" podID="7b23efa1-6306-4879-8d27-2a334daac420" containerID="546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22" exitCode=0
Jul 28 01:33:18.125056 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.124649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fbb6d9f-v7qp5" event={"ID":"7b23efa1-6306-4879-8d27-2a334daac420","Type":"ContainerDied","Data":"546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22"}
Jul 28 01:33:18.125056 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.124683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9fbb6d9f-v7qp5" event={"ID":"7b23efa1-6306-4879-8d27-2a334daac420","Type":"ContainerDied","Data":"6c239da7c8a79ce22e66b0f0ff169b89fbf4bf21231c13c5406c51d4f2c71987"}
Jul 28 01:33:18.125056 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.124701    2579 scope.go:117] "RemoveContainer" containerID="546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22"
Jul 28 01:33:18.125056 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.124650    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9fbb6d9f-v7qp5"
Jul 28 01:33:18.133510 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.133462    2579 scope.go:117] "RemoveContainer" containerID="546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22"
Jul 28 01:33:18.133738 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:33:18.133721    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22\": container with ID starting with 546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22 not found: ID does not exist" containerID="546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22"
Jul 28 01:33:18.133784 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.133748    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22"} err="failed to get container status \"546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22\": rpc error: code = NotFound desc = could not find container \"546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22\": container with ID starting with 546b96eb78a189df51de28f95a88dfe7504879b7e69f7224faead8efdc355e22 not found: ID does not exist"
Jul 28 01:33:18.145659 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.145639    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9fbb6d9f-v7qp5"]
Jul 28 01:33:18.152654 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:18.152635    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9fbb6d9f-v7qp5"]
Jul 28 01:33:19.163683 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:19.163650    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b23efa1-6306-4879-8d27-2a334daac420" path="/var/lib/kubelet/pods/7b23efa1-6306-4879-8d27-2a334daac420/volumes"
Jul 28 01:33:21.010898 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:21.010864    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-wlfjl"
Jul 28 01:33:21.014061 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:21.014041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:33:29.805501 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.805439    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-86vg7"]
Jul 28 01:33:29.805965 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.805670    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-ff96d5959-86vg7" podUID="4677238c-6df9-4ec2-9af2-8c1aad40e7c4" containerName="manager" containerID="cri-o://fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f" gracePeriod=10
Jul 28 01:33:29.845796 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.845766    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bvgb7"]
Jul 28 01:33:29.846089 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.846077    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b23efa1-6306-4879-8d27-2a334daac420" containerName="console"
Jul 28 01:33:29.846134 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.846091    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b23efa1-6306-4879-8d27-2a334daac420" containerName="console"
Jul 28 01:33:29.846168 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.846155    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b23efa1-6306-4879-8d27-2a334daac420" containerName="console"
Jul 28 01:33:29.848016 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.847999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:29.860719 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.860694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bvgb7"]
Jul 28 01:33:29.865507 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.865463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqgj\" (UniqueName: \"kubernetes.io/projected/b96ed6ac-bb97-4df9-8612-ca316deec364-kube-api-access-stqgj\") pod \"kserve-controller-manager-ff96d5959-bvgb7\" (UID: \"b96ed6ac-bb97-4df9-8612-ca316deec364\") " pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:29.865600 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.865585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b96ed6ac-bb97-4df9-8612-ca316deec364-cert\") pod \"kserve-controller-manager-ff96d5959-bvgb7\" (UID: \"b96ed6ac-bb97-4df9-8612-ca316deec364\") " pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:29.966094 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.966064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqgj\" (UniqueName: \"kubernetes.io/projected/b96ed6ac-bb97-4df9-8612-ca316deec364-kube-api-access-stqgj\") pod \"kserve-controller-manager-ff96d5959-bvgb7\" (UID: \"b96ed6ac-bb97-4df9-8612-ca316deec364\") " pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:29.966260 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.966139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b96ed6ac-bb97-4df9-8612-ca316deec364-cert\") pod \"kserve-controller-manager-ff96d5959-bvgb7\" (UID: \"b96ed6ac-bb97-4df9-8612-ca316deec364\") " pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:29.968931 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.968868    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b96ed6ac-bb97-4df9-8612-ca316deec364-cert\") pod \"kserve-controller-manager-ff96d5959-bvgb7\" (UID: \"b96ed6ac-bb97-4df9-8612-ca316deec364\") " pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:29.974735 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:29.974716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqgj\" (UniqueName: \"kubernetes.io/projected/b96ed6ac-bb97-4df9-8612-ca316deec364-kube-api-access-stqgj\") pod \"kserve-controller-manager-ff96d5959-bvgb7\" (UID: \"b96ed6ac-bb97-4df9-8612-ca316deec364\") " pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:30.038769 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.038745    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:33:30.066500 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.066399    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlz9l\" (UniqueName: \"kubernetes.io/projected/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-kube-api-access-hlz9l\") pod \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") "
Jul 28 01:33:30.066500 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.066450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert\") pod \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\" (UID: \"4677238c-6df9-4ec2-9af2-8c1aad40e7c4\") "
Jul 28 01:33:30.069129 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.069099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-kube-api-access-hlz9l" (OuterVolumeSpecName: "kube-api-access-hlz9l") pod "4677238c-6df9-4ec2-9af2-8c1aad40e7c4" (UID: "4677238c-6df9-4ec2-9af2-8c1aad40e7c4"). InnerVolumeSpecName "kube-api-access-hlz9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:33:30.069230 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.069099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert" (OuterVolumeSpecName: "cert") pod "4677238c-6df9-4ec2-9af2-8c1aad40e7c4" (UID: "4677238c-6df9-4ec2-9af2-8c1aad40e7c4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:33:30.167363 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.167333    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlz9l\" (UniqueName: \"kubernetes.io/projected/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-kube-api-access-hlz9l\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:30.167363 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.167356    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4677238c-6df9-4ec2-9af2-8c1aad40e7c4-cert\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:33:30.179915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.179885    2579 generic.go:356] "Generic (PLEG): container finished" podID="4677238c-6df9-4ec2-9af2-8c1aad40e7c4" containerID="fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f" exitCode=0
Jul 28 01:33:30.180029 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.179937    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-86vg7"
Jul 28 01:33:30.180029 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.179971    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-86vg7" event={"ID":"4677238c-6df9-4ec2-9af2-8c1aad40e7c4","Type":"ContainerDied","Data":"fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f"}
Jul 28 01:33:30.180098 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.180029    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-86vg7" event={"ID":"4677238c-6df9-4ec2-9af2-8c1aad40e7c4","Type":"ContainerDied","Data":"0a6f616b6d5a1bce1c254258d9c482096bca97fabc4fb70050ea81f30dace190"}
Jul 28 01:33:30.180098 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.180054    2579 scope.go:117] "RemoveContainer" containerID="fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f"
Jul 28 01:33:30.187633 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.187600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:30.188095 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.188080    2579 scope.go:117] "RemoveContainer" containerID="fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f"
Jul 28 01:33:30.188331 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:33:30.188311    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f\": container with ID starting with fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f not found: ID does not exist" containerID="fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f"
Jul 28 01:33:30.188404 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.188338    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f"} err="failed to get container status \"fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f\": rpc error: code = NotFound desc = could not find container \"fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f\": container with ID starting with fea9d3f2a5945191ed2ddf522fa1210e3b85c345fb1f206b16ced6b6836b123f not found: ID does not exist"
Jul 28 01:33:30.199826 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.199807    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-86vg7"]
Jul 28 01:33:30.205687 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.205667    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-86vg7"]
Jul 28 01:33:30.311577 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:30.311547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-ff96d5959-bvgb7"]
Jul 28 01:33:30.314519 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:33:30.314472    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ed6ac_bb97_4df9_8612_ca316deec364.slice/crio-a235d84b9fa8e48853c2ac572371d434f1692b36cbbc3e0b847c188f76ada338 WatchSource:0}: Error finding container a235d84b9fa8e48853c2ac572371d434f1692b36cbbc3e0b847c188f76ada338: Status 404 returned error can't find the container with id a235d84b9fa8e48853c2ac572371d434f1692b36cbbc3e0b847c188f76ada338
Jul 28 01:33:31.164233 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:31.164193    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4677238c-6df9-4ec2-9af2-8c1aad40e7c4" path="/var/lib/kubelet/pods/4677238c-6df9-4ec2-9af2-8c1aad40e7c4/volumes"
Jul 28 01:33:31.187165 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:31.187137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-bvgb7" event={"ID":"b96ed6ac-bb97-4df9-8612-ca316deec364","Type":"ContainerStarted","Data":"3440b6dc3eb6d7c06b9e2275679aedd0160b8b7614bdc56bcaf17ac4046c93d9"}
Jul 28 01:33:31.187300 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:31.187187    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-ff96d5959-bvgb7" event={"ID":"b96ed6ac-bb97-4df9-8612-ca316deec364","Type":"ContainerStarted","Data":"a235d84b9fa8e48853c2ac572371d434f1692b36cbbc3e0b847c188f76ada338"}
Jul 28 01:33:31.187300 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:31.187291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:33:31.205974 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:33:31.205933    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-ff96d5959-bvgb7" podStartSLOduration=1.7521493910000001 podStartE2EDuration="2.205918678s" totalImagesPullingTime="453.769287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:31.2042986 +0000 UTC m=+470.510787382" watchObservedRunningTime="2026-07-28 01:33:31.205918678 +0000 UTC m=+470.512407458"
Jul 28 01:34:02.196322 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:34:02.196291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-ff96d5959-bvgb7"
Jul 28 01:35:41.068385 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:41.068349    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:35:41.075241 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:41.075221    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:35:50.757577 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.757542    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"]
Jul 28 01:35:50.758067 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.757882    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4677238c-6df9-4ec2-9af2-8c1aad40e7c4" containerName="manager"
Jul 28 01:35:50.758067 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.757896    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4677238c-6df9-4ec2-9af2-8c1aad40e7c4" containerName="manager"
Jul 28 01:35:50.758067 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.757961    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4677238c-6df9-4ec2-9af2-8c1aad40e7c4" containerName="manager"
Jul 28 01:35:50.760989 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.760969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:50.763140 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.763120    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c7854-predictor-serving-cert\""
Jul 28 01:35:50.763382 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.763359    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:35:50.764132 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.764114    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:35:50.764216 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.764137    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c7854-kube-rbac-proxy-sar-config\""
Jul 28 01:35:50.764216 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.764121    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mcfll\""
Jul 28 01:35:50.768254 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.768235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"]
Jul 28 01:35:50.853824 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.853793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:50.853999 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.853831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wl2\" (UniqueName: \"kubernetes.io/projected/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-api-access-x7wl2\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:50.853999 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.853875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:50.954686 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.954651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:50.954686 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.954693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wl2\" (UniqueName: \"kubernetes.io/projected/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-api-access-x7wl2\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:50.954896 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.954735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:50.954896 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:35:50.954801    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c7854-predictor-serving-cert: secret "error-404-isvc-c7854-predictor-serving-cert" not found
Jul 28 01:35:50.954979 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:35:50.954897    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls podName:bcd1b450-ad80-459f-98f7-0eb778b6320a nodeName:}" failed. No retries permitted until 2026-07-28 01:35:51.454863586 +0000 UTC m=+610.761352351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls") pod "error-404-isvc-c7854-predictor-55858f5444-2l7kd" (UID: "bcd1b450-ad80-459f-98f7-0eb778b6320a") : secret "error-404-isvc-c7854-predictor-serving-cert" not found
Jul 28 01:35:50.955386 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.955368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:50.965212 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:50.965185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wl2\" (UniqueName: \"kubernetes.io/projected/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-api-access-x7wl2\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:51.459573 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:51.459538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:51.462044 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:51.462017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls\") pod \"error-404-isvc-c7854-predictor-55858f5444-2l7kd\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") " pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:51.672599 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:51.672564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:35:51.798356 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:51.798334    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"]
Jul 28 01:35:51.800305 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:35:51.800277    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd1b450_ad80_459f_98f7_0eb778b6320a.slice/crio-df793d96ca90bfea7e273b0d7f231360998e5aa0f2ce77f1385445887407d3f9 WatchSource:0}: Error finding container df793d96ca90bfea7e273b0d7f231360998e5aa0f2ce77f1385445887407d3f9: Status 404 returned error can't find the container with id df793d96ca90bfea7e273b0d7f231360998e5aa0f2ce77f1385445887407d3f9
Jul 28 01:35:52.784292 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:35:52.784245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" event={"ID":"bcd1b450-ad80-459f-98f7-0eb778b6320a","Type":"ContainerStarted","Data":"df793d96ca90bfea7e273b0d7f231360998e5aa0f2ce77f1385445887407d3f9"}
Jul 28 01:36:03.942837 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:03.942818    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:36:04.853975 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:04.853941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" event={"ID":"bcd1b450-ad80-459f-98f7-0eb778b6320a","Type":"ContainerStarted","Data":"b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6"}
Jul 28 01:36:06.869610 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:06.869569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" event={"ID":"bcd1b450-ad80-459f-98f7-0eb778b6320a","Type":"ContainerStarted","Data":"806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203"}
Jul 28 01:36:06.869980 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:06.869810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:36:06.869980 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:06.869952    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:36:06.871170 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:06.871144    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:36:06.911298 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:06.911252    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podStartSLOduration=2.528595967 podStartE2EDuration="16.91123763s" totalImagesPullingTime="14.382641663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:06.91000767 +0000 UTC m=+626.216496457" watchObservedRunningTime="2026-07-28 01:36:06.91123763 +0000 UTC m=+626.217726411"
Jul 28 01:36:07.874216 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:07.874178    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:36:12.879954 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:12.879925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:36:12.880399 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:12.880357    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:36:22.880702 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:22.880659    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:36:32.881082 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:32.881035    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:36:42.881206 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:42.881162    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:36:52.881647 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:36:52.881620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:37:20.350288 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.350255    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"]
Jul 28 01:37:20.354928 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.354895    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:20.357685 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.357657    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c7854-kube-rbac-proxy-sar-config\""
Jul 28 01:37:20.357821 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.357686    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c7854-serving-cert\""
Jul 28 01:37:20.361726 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.361704    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"]
Jul 28 01:37:20.427797 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.427764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-openshift-service-ca-bundle\") pod \"switch-graph-c7854-5f88778cb9-czp4v\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") " pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:20.427797 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.427802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls\") pod \"switch-graph-c7854-5f88778cb9-czp4v\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") " pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:20.529223 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.529185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-openshift-service-ca-bundle\") pod \"switch-graph-c7854-5f88778cb9-czp4v\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") " pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:20.529223 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.529226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls\") pod \"switch-graph-c7854-5f88778cb9-czp4v\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") " pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:20.529413 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:37:20.529321    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-c7854-serving-cert: secret "switch-graph-c7854-serving-cert" not found
Jul 28 01:37:20.529413 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:37:20.529383    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls podName:adfe9417-6bd9-47e1-b96b-ae5f82f63dea nodeName:}" failed. No retries permitted until 2026-07-28 01:37:21.029367792 +0000 UTC m=+700.335856551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls") pod "switch-graph-c7854-5f88778cb9-czp4v" (UID: "adfe9417-6bd9-47e1-b96b-ae5f82f63dea") : secret "switch-graph-c7854-serving-cert" not found
Jul 28 01:37:20.529858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:20.529840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-openshift-service-ca-bundle\") pod \"switch-graph-c7854-5f88778cb9-czp4v\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") " pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:21.034282 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:21.034245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls\") pod \"switch-graph-c7854-5f88778cb9-czp4v\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") " pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:21.036569 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:21.036544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls\") pod \"switch-graph-c7854-5f88778cb9-czp4v\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") " pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:21.266165 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:21.266131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:21.390027 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:21.390004    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"]
Jul 28 01:37:21.392165 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:37:21.392126    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfe9417_6bd9_47e1_b96b_ae5f82f63dea.slice/crio-0c05c2db9281aaef483611a2133f811906148eb907fd7bd000100293b9dfe903 WatchSource:0}: Error finding container 0c05c2db9281aaef483611a2133f811906148eb907fd7bd000100293b9dfe903: Status 404 returned error can't find the container with id 0c05c2db9281aaef483611a2133f811906148eb907fd7bd000100293b9dfe903
Jul 28 01:37:22.189819 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:22.189771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" event={"ID":"adfe9417-6bd9-47e1-b96b-ae5f82f63dea","Type":"ContainerStarted","Data":"0c05c2db9281aaef483611a2133f811906148eb907fd7bd000100293b9dfe903"}
Jul 28 01:37:24.199909 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:24.199875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" event={"ID":"adfe9417-6bd9-47e1-b96b-ae5f82f63dea","Type":"ContainerStarted","Data":"9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2"}
Jul 28 01:37:24.200265 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:24.200012    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:24.215801 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:24.215759    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" podStartSLOduration=2.316527853 podStartE2EDuration="4.215745552s" totalImagesPullingTime="1.899217699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:24.21486337 +0000 UTC m=+703.521352150" watchObservedRunningTime="2026-07-28 01:37:24.215745552 +0000 UTC m=+703.522234333"
Jul 28 01:37:30.209456 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.209426    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:30.555789 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.555718    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"]
Jul 28 01:37:30.555922 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.555888    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854" containerID="cri-o://9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2" gracePeriod=30
Jul 28 01:37:30.829213 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.829138    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"]
Jul 28 01:37:30.832207 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.832187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:30.834376 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.834354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-33843-predictor-serving-cert\""
Jul 28 01:37:30.834589 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.834571    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-33843-kube-rbac-proxy-sar-config\""
Jul 28 01:37:30.841325 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.841302    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"]
Jul 28 01:37:30.885935 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.885907    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"]
Jul 28 01:37:30.886131 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.886109    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container" containerID="cri-o://b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6" gracePeriod=30
Jul 28 01:37:30.886220 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.886141    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kube-rbac-proxy" containerID="cri-o://806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203" gracePeriod=30
Jul 28 01:37:30.923260 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.923234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:30.923357 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.923271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29lbq\" (UniqueName: \"kubernetes.io/projected/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-api-access-29lbq\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:30.923410 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:30.923368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.023978 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.023948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.023978 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.023980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29lbq\" (UniqueName: \"kubernetes.io/projected/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-api-access-29lbq\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.024165 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.024032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.024165 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:37:31.024085    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-33843-predictor-serving-cert: secret "error-404-isvc-33843-predictor-serving-cert" not found
Jul 28 01:37:31.024165 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:37:31.024138    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls podName:34158f2a-5b02-4b81-8a01-c3ee8016668b nodeName:}" failed. No retries permitted until 2026-07-28 01:37:31.524123585 +0000 UTC m=+710.830612344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls") pod "error-404-isvc-33843-predictor-77ccc76b96-66hdc" (UID: "34158f2a-5b02-4b81-8a01-c3ee8016668b") : secret "error-404-isvc-33843-predictor-serving-cert" not found
Jul 28 01:37:31.024602 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.024586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.034590 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.034567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29lbq\" (UniqueName: \"kubernetes.io/projected/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-api-access-29lbq\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.234199 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.234162    2579 generic.go:356] "Generic (PLEG): container finished" podID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerID="806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203" exitCode=2
Jul 28 01:37:31.234583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.234215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" event={"ID":"bcd1b450-ad80-459f-98f7-0eb778b6320a","Type":"ContainerDied","Data":"806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203"}
Jul 28 01:37:31.529506 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.529401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.531745 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.531714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls\") pod \"error-404-isvc-33843-predictor-77ccc76b96-66hdc\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") " pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.742315 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.742279    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:31.868546 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:31.868518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"]
Jul 28 01:37:31.870550 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:37:31.870524    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34158f2a_5b02_4b81_8a01_c3ee8016668b.slice/crio-69a80e0b3448b88456c41a3632fb183523027e22f4e43c8c074edaec4c85d6f1 WatchSource:0}: Error finding container 69a80e0b3448b88456c41a3632fb183523027e22f4e43c8c074edaec4c85d6f1: Status 404 returned error can't find the container with id 69a80e0b3448b88456c41a3632fb183523027e22f4e43c8c074edaec4c85d6f1
Jul 28 01:37:32.240159 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.240125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" event={"ID":"34158f2a-5b02-4b81-8a01-c3ee8016668b","Type":"ContainerStarted","Data":"e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512"}
Jul 28 01:37:32.240580 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.240208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" event={"ID":"34158f2a-5b02-4b81-8a01-c3ee8016668b","Type":"ContainerStarted","Data":"b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66"}
Jul 28 01:37:32.240580 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.240232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" event={"ID":"34158f2a-5b02-4b81-8a01-c3ee8016668b","Type":"ContainerStarted","Data":"69a80e0b3448b88456c41a3632fb183523027e22f4e43c8c074edaec4c85d6f1"}
Jul 28 01:37:32.240580 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.240257    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:32.240580 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.240280    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:32.241802 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.241775    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:37:32.257453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.257413    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podStartSLOduration=2.257400688 podStartE2EDuration="2.257400688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:32.256413766 +0000 UTC m=+711.562902570" watchObservedRunningTime="2026-07-28 01:37:32.257400688 +0000 UTC m=+711.563889518"
Jul 28 01:37:32.874685 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.874591    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 01:37:32.881138 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:32.881062    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 01:37:33.244890 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:33.244804    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:37:34.137697 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.137674    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:37:34.250184 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250105    2579 generic.go:356] "Generic (PLEG): container finished" podID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerID="b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6" exitCode=0
Jul 28 01:37:34.250184 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" event={"ID":"bcd1b450-ad80-459f-98f7-0eb778b6320a","Type":"ContainerDied","Data":"b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6"}
Jul 28 01:37:34.250184 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250163    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"
Jul 28 01:37:34.250666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd" event={"ID":"bcd1b450-ad80-459f-98f7-0eb778b6320a","Type":"ContainerDied","Data":"df793d96ca90bfea7e273b0d7f231360998e5aa0f2ce77f1385445887407d3f9"}
Jul 28 01:37:34.250666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250292    2579 scope.go:117] "RemoveContainer" containerID="806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203"
Jul 28 01:37:34.250666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250441    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls\") pod \"bcd1b450-ad80-459f-98f7-0eb778b6320a\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") "
Jul 28 01:37:34.250666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-rbac-proxy-sar-config\") pod \"bcd1b450-ad80-459f-98f7-0eb778b6320a\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") "
Jul 28 01:37:34.250666 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7wl2\" (UniqueName: \"kubernetes.io/projected/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-api-access-x7wl2\") pod \"bcd1b450-ad80-459f-98f7-0eb778b6320a\" (UID: \"bcd1b450-ad80-459f-98f7-0eb778b6320a\") "
Jul 28 01:37:34.250915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.250880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bcd1b450-ad80-459f-98f7-0eb778b6320a" (UID: "bcd1b450-ad80-459f-98f7-0eb778b6320a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:37:34.252899 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.252870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bcd1b450-ad80-459f-98f7-0eb778b6320a" (UID: "bcd1b450-ad80-459f-98f7-0eb778b6320a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:37:34.252986 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.252895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-api-access-x7wl2" (OuterVolumeSpecName: "kube-api-access-x7wl2") pod "bcd1b450-ad80-459f-98f7-0eb778b6320a" (UID: "bcd1b450-ad80-459f-98f7-0eb778b6320a"). InnerVolumeSpecName "kube-api-access-x7wl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:37:34.262950 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.262931    2579 scope.go:117] "RemoveContainer" containerID="b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6"
Jul 28 01:37:34.269768 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.269750    2579 scope.go:117] "RemoveContainer" containerID="806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203"
Jul 28 01:37:34.270004 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:37:34.269985    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203\": container with ID starting with 806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203 not found: ID does not exist" containerID="806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203"
Jul 28 01:37:34.270074 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.270017    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203"} err="failed to get container status \"806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203\": rpc error: code = NotFound desc = could not find container \"806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203\": container with ID starting with 806a5ac3ca9ae6beaa06abb0434474d1573b25eb10ca908fa90fa60bfd386203 not found: ID does not exist"
Jul 28 01:37:34.270074 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.270040    2579 scope.go:117] "RemoveContainer" containerID="b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6"
Jul 28 01:37:34.270286 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:37:34.270266    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6\": container with ID starting with b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6 not found: ID does not exist" containerID="b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6"
Jul 28 01:37:34.270331 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.270293    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6"} err="failed to get container status \"b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6\": rpc error: code = NotFound desc = could not find container \"b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6\": container with ID starting with b707c12069934392ded028a3235b8422b9e936be8b93f07200420d8587a2e3f6 not found: ID does not exist"
Jul 28 01:37:34.352007 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.351972    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bcd1b450-ad80-459f-98f7-0eb778b6320a-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:37:34.352007 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.352003    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:37:34.352188 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.352017    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wl2\" (UniqueName: \"kubernetes.io/projected/bcd1b450-ad80-459f-98f7-0eb778b6320a-kube-api-access-x7wl2\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:37:34.572006 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.571978    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"]
Jul 28 01:37:34.573474 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:34.573456    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7854-predictor-55858f5444-2l7kd"]
Jul 28 01:37:35.163781 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:35.163751    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" path="/var/lib/kubelet/pods/bcd1b450-ad80-459f-98f7-0eb778b6320a/volumes"
Jul 28 01:37:35.207974 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:35.207942    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:38.249920 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:38.249891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:37:38.250468 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:38.250440    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:37:40.208596 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:40.208551    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:45.209079 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:45.209031    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:45.209591 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:45.209142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:37:48.250663 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:48.250624    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:37:50.208329 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:50.208282    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:55.209064 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:55.209025    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:37:58.251062 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:37:58.251023    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:38:00.208043 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.207997    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:00.321363 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.321319    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"]
Jul 28 01:38:00.321870 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.321853    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container"
Jul 28 01:38:00.321928 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.321875    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container"
Jul 28 01:38:00.321928 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.321890    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kube-rbac-proxy"
Jul 28 01:38:00.321928 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.321899    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kube-rbac-proxy"
Jul 28 01:38:00.322031 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.322006    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kserve-container"
Jul 28 01:38:00.322031 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.322018    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcd1b450-ad80-459f-98f7-0eb778b6320a" containerName="kube-rbac-proxy"
Jul 28 01:38:00.325171 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.325155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:00.327508 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.327475    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jul 28 01:38:00.327583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.327475    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jul 28 01:38:00.334517 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.334476    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"]
Jul 28 01:38:00.371453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.371424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-vfd7b\" (UID: \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:00.371453 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.371452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-proxy-tls\") pod \"model-chainer-b84f4c8c7-vfd7b\" (UID: \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:00.472166 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.472085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-vfd7b\" (UID: \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:00.472166 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.472121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-proxy-tls\") pod \"model-chainer-b84f4c8c7-vfd7b\" (UID: \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:00.472735 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.472703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-openshift-service-ca-bundle\") pod \"model-chainer-b84f4c8c7-vfd7b\" (UID: \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:00.474730 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.474707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-proxy-tls\") pod \"model-chainer-b84f4c8c7-vfd7b\" (UID: \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\") " pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:00.635920 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.635892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:00.695662 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.695638    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:38:00.774973 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.774848    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"]
Jul 28 01:38:00.775047 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.775032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-openshift-service-ca-bundle\") pod \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") "
Jul 28 01:38:00.775151 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.775117    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls\") pod \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\" (UID: \"adfe9417-6bd9-47e1-b96b-ae5f82f63dea\") "
Jul 28 01:38:00.775368 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.775345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "adfe9417-6bd9-47e1-b96b-ae5f82f63dea" (UID: "adfe9417-6bd9-47e1-b96b-ae5f82f63dea"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:00.777079 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.777056    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "adfe9417-6bd9-47e1-b96b-ae5f82f63dea" (UID: "adfe9417-6bd9-47e1-b96b-ae5f82f63dea"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:38:00.777198 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:38:00.777099    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0ca180_044d_45d0_b0f1_63ebeba9d2e1.slice/crio-7434e5521b7b517ed76d5c336397a9ab88d9cc8eaeb08928ed6d545085b120d4 WatchSource:0}: Error finding container 7434e5521b7b517ed76d5c336397a9ab88d9cc8eaeb08928ed6d545085b120d4: Status 404 returned error can't find the container with id 7434e5521b7b517ed76d5c336397a9ab88d9cc8eaeb08928ed6d545085b120d4
Jul 28 01:38:00.876181 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.876148    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-openshift-service-ca-bundle\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:38:00.876181 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:00.876171    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfe9417-6bd9-47e1-b96b-ae5f82f63dea-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:38:01.366575 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.366538    2579 generic.go:356] "Generic (PLEG): container finished" podID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerID="9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2" exitCode=0
Jul 28 01:38:01.366996 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.366584    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"
Jul 28 01:38:01.366996 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.366623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" event={"ID":"adfe9417-6bd9-47e1-b96b-ae5f82f63dea","Type":"ContainerDied","Data":"9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2"}
Jul 28 01:38:01.366996 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.366669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v" event={"ID":"adfe9417-6bd9-47e1-b96b-ae5f82f63dea","Type":"ContainerDied","Data":"0c05c2db9281aaef483611a2133f811906148eb907fd7bd000100293b9dfe903"}
Jul 28 01:38:01.366996 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.366691    2579 scope.go:117] "RemoveContainer" containerID="9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2"
Jul 28 01:38:01.368455 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.368432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" event={"ID":"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1","Type":"ContainerStarted","Data":"bd7a9c1676512387fae203153626c79c3ad2ee95a24c185de982372093a290b8"}
Jul 28 01:38:01.368583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.368507    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:01.368583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.368532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" event={"ID":"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1","Type":"ContainerStarted","Data":"7434e5521b7b517ed76d5c336397a9ab88d9cc8eaeb08928ed6d545085b120d4"}
Jul 28 01:38:01.374813 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.374782    2579 scope.go:117] "RemoveContainer" containerID="9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2"
Jul 28 01:38:01.375071 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:38:01.375049    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2\": container with ID starting with 9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2 not found: ID does not exist" containerID="9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2"
Jul 28 01:38:01.375131 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.375078    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2"} err="failed to get container status \"9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2\": rpc error: code = NotFound desc = could not find container \"9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2\": container with ID starting with 9f371436dd1ea5d6220bf4cf6b96a02598f20c71d2bd6773793e9e9d4161e8b2 not found: ID does not exist"
Jul 28 01:38:01.382806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.382774    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"]
Jul 28 01:38:01.388969 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.388949    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c7854-5f88778cb9-czp4v"]
Jul 28 01:38:01.401881 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:01.401840    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" podStartSLOduration=1.4018265269999999 podStartE2EDuration="1.401826527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:01.401174317 +0000 UTC m=+740.707663133" watchObservedRunningTime="2026-07-28 01:38:01.401826527 +0000 UTC m=+740.708315362"
Jul 28 01:38:03.163281 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:03.163240    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" path="/var/lib/kubelet/pods/adfe9417-6bd9-47e1-b96b-ae5f82f63dea/volumes"
Jul 28 01:38:07.378933 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:07.378907    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:08.251389 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:08.251352    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:38:10.405144 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.405102    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"]
Jul 28 01:38:10.405520 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.405406    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer" containerID="cri-o://bd7a9c1676512387fae203153626c79c3ad2ee95a24c185de982372093a290b8" gracePeriod=30
Jul 28 01:38:10.742848 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.742772    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"]
Jul 28 01:38:10.743142 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.743128    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854"
Jul 28 01:38:10.743142 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.743143    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854"
Jul 28 01:38:10.743231 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.743199    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfe9417-6bd9-47e1-b96b-ae5f82f63dea" containerName="switch-graph-c7854"
Jul 28 01:38:10.746227 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.746213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.748348 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.748327    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3d51c-predictor-serving-cert\""
Jul 28 01:38:10.748747 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.748728    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3d51c-kube-rbac-proxy-sar-config\""
Jul 28 01:38:10.761344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.761320    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"]
Jul 28 01:38:10.860674 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.860642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhqm\" (UniqueName: \"kubernetes.io/projected/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-api-access-bzhqm\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.860839 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.860684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.860839 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.860725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/635abc6b-b5be-420a-9cca-55974f2e12f0-proxy-tls\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.961764 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.961730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhqm\" (UniqueName: \"kubernetes.io/projected/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-api-access-bzhqm\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.961764 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.961770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.961966 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.961805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/635abc6b-b5be-420a-9cca-55974f2e12f0-proxy-tls\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.962628 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.962602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.964151 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.964135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/635abc6b-b5be-420a-9cca-55974f2e12f0-proxy-tls\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:10.970536 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:10.970508    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhqm\" (UniqueName: \"kubernetes.io/projected/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-api-access-bzhqm\") pod \"error-404-isvc-3d51c-predictor-56c57c964b-rtd7l\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") " pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:11.056342 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:11.056257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:11.184255 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:11.184234    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"]
Jul 28 01:38:11.186738 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:38:11.186712    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635abc6b_b5be_420a_9cca_55974f2e12f0.slice/crio-9e0ccc2ac1b6d5f63fd7ab7d007b92818cd2181f01a598f2e5360ed25b0b3bec WatchSource:0}: Error finding container 9e0ccc2ac1b6d5f63fd7ab7d007b92818cd2181f01a598f2e5360ed25b0b3bec: Status 404 returned error can't find the container with id 9e0ccc2ac1b6d5f63fd7ab7d007b92818cd2181f01a598f2e5360ed25b0b3bec
Jul 28 01:38:11.418795 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:11.418763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" event={"ID":"635abc6b-b5be-420a-9cca-55974f2e12f0","Type":"ContainerStarted","Data":"0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89"}
Jul 28 01:38:11.419128 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:11.418825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" event={"ID":"635abc6b-b5be-420a-9cca-55974f2e12f0","Type":"ContainerStarted","Data":"0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b"}
Jul 28 01:38:11.419128 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:11.418847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" event={"ID":"635abc6b-b5be-420a-9cca-55974f2e12f0","Type":"ContainerStarted","Data":"9e0ccc2ac1b6d5f63fd7ab7d007b92818cd2181f01a598f2e5360ed25b0b3bec"}
Jul 28 01:38:11.419128 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:11.418896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:11.439515 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:11.439449    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podStartSLOduration=1.439436334 podStartE2EDuration="1.439436334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:11.438125033 +0000 UTC m=+750.744613833" watchObservedRunningTime="2026-07-28 01:38:11.439436334 +0000 UTC m=+750.745925115"
Jul 28 01:38:12.378131 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:12.378092    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:12.424169 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:12.424135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:12.425566 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:12.425539    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:38:13.428264 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:13.428224    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:38:17.377700 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:17.377657    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:18.252251 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:18.252222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:38:18.432855 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:18.432829    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:38:18.433320 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:18.433291    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:38:22.377431 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:22.377390    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:22.377892 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:22.377552    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:27.377983 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:27.377943    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:28.433843 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:28.433810    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:38:32.377715 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:32.377677    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:37.377616 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:37.377574    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:38:38.434000 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:38.433964    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:38:40.552688 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.552656    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerID="bd7a9c1676512387fae203153626c79c3ad2ee95a24c185de982372093a290b8" exitCode=0
Jul 28 01:38:40.552989 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.552700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" event={"ID":"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1","Type":"ContainerDied","Data":"bd7a9c1676512387fae203153626c79c3ad2ee95a24c185de982372093a290b8"}
Jul 28 01:38:40.552989 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.552759    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b" event={"ID":"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1","Type":"ContainerDied","Data":"7434e5521b7b517ed76d5c336397a9ab88d9cc8eaeb08928ed6d545085b120d4"}
Jul 28 01:38:40.552989 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.552775    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7434e5521b7b517ed76d5c336397a9ab88d9cc8eaeb08928ed6d545085b120d4"
Jul 28 01:38:40.554823 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.554807    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:40.603617 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.603587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-openshift-service-ca-bundle\") pod \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\" (UID: \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\") "
Jul 28 01:38:40.603757 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.603674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-proxy-tls\") pod \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\" (UID: \"8c0ca180-044d-45d0-b0f1-63ebeba9d2e1\") "
Jul 28 01:38:40.604046 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.604015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" (UID: "8c0ca180-044d-45d0-b0f1-63ebeba9d2e1"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:40.605665 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.605644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" (UID: "8c0ca180-044d-45d0-b0f1-63ebeba9d2e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:38:40.704201 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.704113    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-openshift-service-ca-bundle\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:38:40.704201 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.704146    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:38:40.798429 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.798399    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"]
Jul 28 01:38:40.798772 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.798750    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer"
Jul 28 01:38:40.798772 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.798765    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer"
Jul 28 01:38:40.798911 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.798832    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" containerName="model-chainer"
Jul 28 01:38:40.801770 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.801747    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:40.806618 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.804064    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-33843-serving-cert\""
Jul 28 01:38:40.806618 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.804364    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-33843-kube-rbac-proxy-sar-config\""
Jul 28 01:38:40.806618 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.805187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-openshift-service-ca-bundle\") pod \"switch-graph-33843-84bc469c8d-t78gn\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") " pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:40.806618 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.805262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls\") pod \"switch-graph-33843-84bc469c8d-t78gn\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") " pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:40.812157 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.812134    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"]
Jul 28 01:38:40.906026 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.905990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls\") pod \"switch-graph-33843-84bc469c8d-t78gn\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") " pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:40.906193 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.906066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-openshift-service-ca-bundle\") pod \"switch-graph-33843-84bc469c8d-t78gn\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") " pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:40.906193 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:38:40.906179    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-33843-serving-cert: secret "switch-graph-33843-serving-cert" not found
Jul 28 01:38:40.906298 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:38:40.906284    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls podName:0d8ae9df-549f-41ea-9e81-7860ac4a4cb8 nodeName:}" failed. No retries permitted until 2026-07-28 01:38:41.406258828 +0000 UTC m=+780.712747608 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls") pod "switch-graph-33843-84bc469c8d-t78gn" (UID: "0d8ae9df-549f-41ea-9e81-7860ac4a4cb8") : secret "switch-graph-33843-serving-cert" not found
Jul 28 01:38:40.906662 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:40.906646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-openshift-service-ca-bundle\") pod \"switch-graph-33843-84bc469c8d-t78gn\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") " pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:41.410575 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:41.410466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls\") pod \"switch-graph-33843-84bc469c8d-t78gn\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") " pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:41.412838 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:41.412810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls\") pod \"switch-graph-33843-84bc469c8d-t78gn\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") " pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:41.418683 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:41.418660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:41.545390 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:41.545360    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"]
Jul 28 01:38:41.548079 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:38:41.548055    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8ae9df_549f_41ea_9e81_7860ac4a4cb8.slice/crio-278696dac9f1b8fdd86f98b66072cd217e43fadba12aab44042864894404a6ae WatchSource:0}: Error finding container 278696dac9f1b8fdd86f98b66072cd217e43fadba12aab44042864894404a6ae: Status 404 returned error can't find the container with id 278696dac9f1b8fdd86f98b66072cd217e43fadba12aab44042864894404a6ae
Jul 28 01:38:41.557947 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:41.557920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" event={"ID":"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8","Type":"ContainerStarted","Data":"278696dac9f1b8fdd86f98b66072cd217e43fadba12aab44042864894404a6ae"}
Jul 28 01:38:41.557947 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:41.557939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"
Jul 28 01:38:41.574899 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:41.574875    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"]
Jul 28 01:38:41.578329 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:41.578309    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-b84f4c8c7-vfd7b"]
Jul 28 01:38:42.564625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:42.564591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" event={"ID":"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8","Type":"ContainerStarted","Data":"f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a"}
Jul 28 01:38:42.565016 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:42.564690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:42.580843 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:42.580796    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" podStartSLOduration=2.580780495 podStartE2EDuration="2.580780495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:42.579903087 +0000 UTC m=+781.886391869" watchObservedRunningTime="2026-07-28 01:38:42.580780495 +0000 UTC m=+781.887269275"
Jul 28 01:38:43.163497 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:43.163459    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0ca180-044d-45d0-b0f1-63ebeba9d2e1" path="/var/lib/kubelet/pods/8c0ca180-044d-45d0-b0f1-63ebeba9d2e1/volumes"
Jul 28 01:38:48.433979 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:48.433944    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:38:48.575268 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:48.575231    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:38:58.434646 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:38:58.434620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:40:41.096585 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:40:41.096516    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:40:41.101256 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:40:41.101233    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:44:41.159407 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:44:41.159372    2579 scope.go:117] "RemoveContainer" containerID="bd7a9c1676512387fae203153626c79c3ad2ee95a24c185de982372093a290b8"
Jul 28 01:45:41.125840 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:45:41.125813    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:45:41.128497 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:45:41.128456    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:46:55.381300 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.381269    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"]
Jul 28 01:46:55.383688 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.381461    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843" containerID="cri-o://f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a" gracePeriod=30
Jul 28 01:46:55.672041 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.671963    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"]
Jul 28 01:46:55.675388 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.675371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.677932 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.677914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-02c81-predictor-serving-cert\""
Jul 28 01:46:55.678773 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.678759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-02c81-kube-rbac-proxy-sar-config\""
Jul 28 01:46:55.687885 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.687865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"]
Jul 28 01:46:55.733305 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.733276    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"]
Jul 28 01:46:55.733545 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.733518    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container" containerID="cri-o://b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66" gracePeriod=30
Jul 28 01:46:55.733636 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.733544    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kube-rbac-proxy" containerID="cri-o://e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512" gracePeriod=30
Jul 28 01:46:55.761363 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.761334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b436056-36ce-43e0-b251-495797c91744-proxy-tls\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.761470 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.761448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9tk8\" (UniqueName: \"kubernetes.io/projected/8b436056-36ce-43e0-b251-495797c91744-kube-api-access-v9tk8\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.761555 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.761518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b436056-36ce-43e0-b251-495797c91744-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.862892 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.862861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9tk8\" (UniqueName: \"kubernetes.io/projected/8b436056-36ce-43e0-b251-495797c91744-kube-api-access-v9tk8\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.863055 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.862907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b436056-36ce-43e0-b251-495797c91744-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.863055 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.862941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b436056-36ce-43e0-b251-495797c91744-proxy-tls\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.863614 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.863594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b436056-36ce-43e0-b251-495797c91744-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.865224 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.865203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b436056-36ce-43e0-b251-495797c91744-proxy-tls\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.872133 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.872113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9tk8\" (UniqueName: \"kubernetes.io/projected/8b436056-36ce-43e0-b251-495797c91744-kube-api-access-v9tk8\") pod \"error-404-isvc-02c81-predictor-58cbbbb866-f6tf6\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") " pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:55.985927 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:55.985844    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:56.116826 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:56.116800    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"]
Jul 28 01:46:56.118850 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:46:56.118819    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b436056_36ce_43e0_b251_495797c91744.slice/crio-cf37e1f77d68abf5b0c8d1a9fb8243549d08ea9ada12281c261d9c1e77549b89 WatchSource:0}: Error finding container cf37e1f77d68abf5b0c8d1a9fb8243549d08ea9ada12281c261d9c1e77549b89: Status 404 returned error can't find the container with id cf37e1f77d68abf5b0c8d1a9fb8243549d08ea9ada12281c261d9c1e77549b89
Jul 28 01:46:56.721164 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:56.721133    2579 generic.go:356] "Generic (PLEG): container finished" podID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerID="e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512" exitCode=2
Jul 28 01:46:56.721578 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:56.721202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" event={"ID":"34158f2a-5b02-4b81-8a01-c3ee8016668b","Type":"ContainerDied","Data":"e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512"}
Jul 28 01:46:56.723013 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:56.722986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" event={"ID":"8b436056-36ce-43e0-b251-495797c91744","Type":"ContainerStarted","Data":"3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d"}
Jul 28 01:46:56.723128 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:56.723042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" event={"ID":"8b436056-36ce-43e0-b251-495797c91744","Type":"ContainerStarted","Data":"c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3"}
Jul 28 01:46:56.723128 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:56.723085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" event={"ID":"8b436056-36ce-43e0-b251-495797c91744","Type":"ContainerStarted","Data":"cf37e1f77d68abf5b0c8d1a9fb8243549d08ea9ada12281c261d9c1e77549b89"}
Jul 28 01:46:56.723205 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:56.723154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:56.740242 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:56.740203    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podStartSLOduration=1.740190262 podStartE2EDuration="1.740190262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:46:56.739324925 +0000 UTC m=+1276.045813744" watchObservedRunningTime="2026-07-28 01:46:56.740190262 +0000 UTC m=+1276.046679104"
Jul 28 01:46:57.727717 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:57.727687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:46:57.729196 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:57.729167    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 01:46:58.245525 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:58.245461    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 28 01:46:58.250808 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:58.250784    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 01:46:58.573944 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:58.573863    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:46:58.731948 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:58.731904    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 01:46:58.984423 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:58.984399    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:46:59.089712 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.089672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-rbac-proxy-sar-config\") pod \"34158f2a-5b02-4b81-8a01-c3ee8016668b\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") "
Jul 28 01:46:59.089880 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.089734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29lbq\" (UniqueName: \"kubernetes.io/projected/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-api-access-29lbq\") pod \"34158f2a-5b02-4b81-8a01-c3ee8016668b\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") "
Jul 28 01:46:59.089880 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.089831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls\") pod \"34158f2a-5b02-4b81-8a01-c3ee8016668b\" (UID: \"34158f2a-5b02-4b81-8a01-c3ee8016668b\") "
Jul 28 01:46:59.090125 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.090099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34158f2a-5b02-4b81-8a01-c3ee8016668b" (UID: "34158f2a-5b02-4b81-8a01-c3ee8016668b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:46:59.091861 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.091837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-api-access-29lbq" (OuterVolumeSpecName: "kube-api-access-29lbq") pod "34158f2a-5b02-4b81-8a01-c3ee8016668b" (UID: "34158f2a-5b02-4b81-8a01-c3ee8016668b"). InnerVolumeSpecName "kube-api-access-29lbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:46:59.091942 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.091844    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34158f2a-5b02-4b81-8a01-c3ee8016668b" (UID: "34158f2a-5b02-4b81-8a01-c3ee8016668b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:46:59.196620 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.191355    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:46:59.196620 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.191455    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29lbq\" (UniqueName: \"kubernetes.io/projected/34158f2a-5b02-4b81-8a01-c3ee8016668b-kube-api-access-29lbq\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:46:59.196620 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.191532    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34158f2a-5b02-4b81-8a01-c3ee8016668b-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:46:59.737402 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.737369    2579 generic.go:356] "Generic (PLEG): container finished" podID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerID="b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66" exitCode=0
Jul 28 01:46:59.737806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.737433    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"
Jul 28 01:46:59.737806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.737449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" event={"ID":"34158f2a-5b02-4b81-8a01-c3ee8016668b","Type":"ContainerDied","Data":"b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66"}
Jul 28 01:46:59.737806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.737556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc" event={"ID":"34158f2a-5b02-4b81-8a01-c3ee8016668b","Type":"ContainerDied","Data":"69a80e0b3448b88456c41a3632fb183523027e22f4e43c8c074edaec4c85d6f1"}
Jul 28 01:46:59.737806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.737587    2579 scope.go:117] "RemoveContainer" containerID="e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512"
Jul 28 01:46:59.746726 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.746696    2579 scope.go:117] "RemoveContainer" containerID="b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66"
Jul 28 01:46:59.752969 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.752949    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"]
Jul 28 01:46:59.754539 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.754516    2579 scope.go:117] "RemoveContainer" containerID="e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512"
Jul 28 01:46:59.754836 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:46:59.754818    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512\": container with ID starting with e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512 not found: ID does not exist" containerID="e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512"
Jul 28 01:46:59.754899 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.754845    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512"} err="failed to get container status \"e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512\": rpc error: code = NotFound desc = could not find container \"e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512\": container with ID starting with e3e9611fc20917bd983e99baa5bf60cf9dc08de02c8008268a140460da779512 not found: ID does not exist"
Jul 28 01:46:59.754899 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.754864    2579 scope.go:117] "RemoveContainer" containerID="b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66"
Jul 28 01:46:59.755129 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:46:59.755107    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66\": container with ID starting with b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66 not found: ID does not exist" containerID="b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66"
Jul 28 01:46:59.755170 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.755138    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66"} err="failed to get container status \"b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66\": rpc error: code = NotFound desc = could not find container \"b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66\": container with ID starting with b2da3bb66e596854ad6c4f0d112bacd4b068f934596bcae8ef663519694ebd66 not found: ID does not exist"
Jul 28 01:46:59.755468 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:46:59.755454    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-33843-predictor-77ccc76b96-66hdc"]
Jul 28 01:47:01.163413 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:01.163381    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" path="/var/lib/kubelet/pods/34158f2a-5b02-4b81-8a01-c3ee8016668b/volumes"
Jul 28 01:47:03.573690 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:03.573654    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:03.736946 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:03.736921    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:47:03.737385 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:03.737356    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 01:47:08.573387 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:08.573341    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:08.573790 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:08.573459    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:47:13.573720 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:13.573676    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:13.737961 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:13.737921    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 01:47:18.573675 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:18.573638    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:23.573650 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:23.573607    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:23.738022 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:23.737985    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 01:47:25.318029 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.317997    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"]
Jul 28 01:47:25.318467 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.318316    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kube-rbac-proxy" containerID="cri-o://0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89" gracePeriod=30
Jul 28 01:47:25.318467 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.318295    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container" containerID="cri-o://0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b" gracePeriod=30
Jul 28 01:47:25.448320 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.448288    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"]
Jul 28 01:47:25.448687 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.448671    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container"
Jul 28 01:47:25.448687 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.448686    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container"
Jul 28 01:47:25.448832 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.448696    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kube-rbac-proxy"
Jul 28 01:47:25.448832 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.448702    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kube-rbac-proxy"
Jul 28 01:47:25.448832 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.448765    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kube-rbac-proxy"
Jul 28 01:47:25.448832 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.448772    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34158f2a-5b02-4b81-8a01-c3ee8016668b" containerName="kserve-container"
Jul 28 01:47:25.451738 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.451717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.453918 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.453894    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2a946-predictor-serving-cert\""
Jul 28 01:47:25.454044 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.454025    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2a946-kube-rbac-proxy-sar-config\""
Jul 28 01:47:25.462067 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.462036    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"]
Jul 28 01:47:25.518173 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.518135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.518335 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.518209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.518408 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.518328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8bq\" (UniqueName: \"kubernetes.io/projected/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-api-access-kg8bq\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.572396 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.572334    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:47:25.619428 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.619400    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-openshift-service-ca-bundle\") pod \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") "
Jul 28 01:47:25.619601 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.619477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls\") pod \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\" (UID: \"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8\") "
Jul 28 01:47:25.619677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.619609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.619677 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.619666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8bq\" (UniqueName: \"kubernetes.io/projected/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-api-access-kg8bq\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.619779 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.619729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.619779 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.619740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" (UID: "0d8ae9df-549f-41ea-9e81-7860ac4a4cb8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:47:25.619885 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:47:25.619790    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-2a946-predictor-serving-cert: secret "error-404-isvc-2a946-predictor-serving-cert" not found
Jul 28 01:47:25.619885 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:47:25.619869    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls podName:c70de991-ec99-4f4a-a9af-44a84b89704b nodeName:}" failed. No retries permitted until 2026-07-28 01:47:26.11984798 +0000 UTC m=+1305.426336753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls") pod "error-404-isvc-2a946-predictor-85b9467f54-9t9hc" (UID: "c70de991-ec99-4f4a-a9af-44a84b89704b") : secret "error-404-isvc-2a946-predictor-serving-cert" not found
Jul 28 01:47:25.620344 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.620322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.621680 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.621661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" (UID: "0d8ae9df-549f-41ea-9e81-7860ac4a4cb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:25.628596 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.628572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8bq\" (UniqueName: \"kubernetes.io/projected/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-api-access-kg8bq\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:25.720726 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.720692    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:47:25.720726 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.720722    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8-openshift-service-ca-bundle\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:47:25.857592 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.857498    2579 generic.go:356] "Generic (PLEG): container finished" podID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerID="f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a" exitCode=0
Jul 28 01:47:25.857592 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.857539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" event={"ID":"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8","Type":"ContainerDied","Data":"f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a"}
Jul 28 01:47:25.857821 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.857552    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"
Jul 28 01:47:25.857821 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.857642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn" event={"ID":"0d8ae9df-549f-41ea-9e81-7860ac4a4cb8","Type":"ContainerDied","Data":"278696dac9f1b8fdd86f98b66072cd217e43fadba12aab44042864894404a6ae"}
Jul 28 01:47:25.857821 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.857674    2579 scope.go:117] "RemoveContainer" containerID="f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a"
Jul 28 01:47:25.859477 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.859452    2579 generic.go:356] "Generic (PLEG): container finished" podID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerID="0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89" exitCode=2
Jul 28 01:47:25.859583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.859510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" event={"ID":"635abc6b-b5be-420a-9cca-55974f2e12f0","Type":"ContainerDied","Data":"0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89"}
Jul 28 01:47:25.866625 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.866606    2579 scope.go:117] "RemoveContainer" containerID="f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a"
Jul 28 01:47:25.866869 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:47:25.866849    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a\": container with ID starting with f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a not found: ID does not exist" containerID="f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a"
Jul 28 01:47:25.866928 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.866882    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a"} err="failed to get container status \"f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a\": rpc error: code = NotFound desc = could not find container \"f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a\": container with ID starting with f162417fb1b9812d0176a1b50707b537deb8a0f54c6c6ca58b5f348ce6879e8a not found: ID does not exist"
Jul 28 01:47:25.877689 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.877669    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"]
Jul 28 01:47:25.885823 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:25.885804    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-33843-84bc469c8d-t78gn"]
Jul 28 01:47:26.124074 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.124045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:26.126380 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.126363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls\") pod \"error-404-isvc-2a946-predictor-85b9467f54-9t9hc\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") " pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:26.362858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.362818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:26.496599 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.496575    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"]
Jul 28 01:47:26.498716 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:47:26.498685    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70de991_ec99_4f4a_a9af_44a84b89704b.slice/crio-940dc72b47df69f86f96594786c24ecc9fa002d056ecbbc7716725b7d22a9c6f WatchSource:0}: Error finding container 940dc72b47df69f86f96594786c24ecc9fa002d056ecbbc7716725b7d22a9c6f: Status 404 returned error can't find the container with id 940dc72b47df69f86f96594786c24ecc9fa002d056ecbbc7716725b7d22a9c6f
Jul 28 01:47:26.866865 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.866832    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" event={"ID":"c70de991-ec99-4f4a-a9af-44a84b89704b","Type":"ContainerStarted","Data":"674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e"}
Jul 28 01:47:26.867038 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.866889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" event={"ID":"c70de991-ec99-4f4a-a9af-44a84b89704b","Type":"ContainerStarted","Data":"3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d"}
Jul 28 01:47:26.867038 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.866914    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" event={"ID":"c70de991-ec99-4f4a-a9af-44a84b89704b","Type":"ContainerStarted","Data":"940dc72b47df69f86f96594786c24ecc9fa002d056ecbbc7716725b7d22a9c6f"}
Jul 28 01:47:26.867038 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.866957    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:26.886018 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:26.885975    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podStartSLOduration=1.885963096 podStartE2EDuration="1.885963096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:47:26.883754015 +0000 UTC m=+1306.190242798" watchObservedRunningTime="2026-07-28 01:47:26.885963096 +0000 UTC m=+1306.192451955"
Jul 28 01:47:27.164455 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:27.164378    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" path="/var/lib/kubelet/pods/0d8ae9df-549f-41ea-9e81-7860ac4a4cb8/volumes"
Jul 28 01:47:27.871897 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:27.871866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:27.873318 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:27.873291    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 01:47:28.428937 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.428889    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 01:47:28.433991 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.433963    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 01:47:28.567676 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.567653    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:47:28.643747 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.643713    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-rbac-proxy-sar-config\") pod \"635abc6b-b5be-420a-9cca-55974f2e12f0\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") "
Jul 28 01:47:28.643909 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.643782    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/635abc6b-b5be-420a-9cca-55974f2e12f0-proxy-tls\") pod \"635abc6b-b5be-420a-9cca-55974f2e12f0\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") "
Jul 28 01:47:28.643909 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.643817    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzhqm\" (UniqueName: \"kubernetes.io/projected/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-api-access-bzhqm\") pod \"635abc6b-b5be-420a-9cca-55974f2e12f0\" (UID: \"635abc6b-b5be-420a-9cca-55974f2e12f0\") "
Jul 28 01:47:28.644079 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.644055    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "635abc6b-b5be-420a-9cca-55974f2e12f0" (UID: "635abc6b-b5be-420a-9cca-55974f2e12f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:47:28.645803 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.645778    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/635abc6b-b5be-420a-9cca-55974f2e12f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "635abc6b-b5be-420a-9cca-55974f2e12f0" (UID: "635abc6b-b5be-420a-9cca-55974f2e12f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:28.645873 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.645849    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-api-access-bzhqm" (OuterVolumeSpecName: "kube-api-access-bzhqm") pod "635abc6b-b5be-420a-9cca-55974f2e12f0" (UID: "635abc6b-b5be-420a-9cca-55974f2e12f0"). InnerVolumeSpecName "kube-api-access-bzhqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:47:28.744398 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.744325    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:47:28.744398 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.744349    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/635abc6b-b5be-420a-9cca-55974f2e12f0-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:47:28.744398 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.744359    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzhqm\" (UniqueName: \"kubernetes.io/projected/635abc6b-b5be-420a-9cca-55974f2e12f0-kube-api-access-bzhqm\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:47:28.877701 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.877662    2579 generic.go:356] "Generic (PLEG): container finished" podID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerID="0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b" exitCode=0
Jul 28 01:47:28.878141 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.877715    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"
Jul 28 01:47:28.878141 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.877745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" event={"ID":"635abc6b-b5be-420a-9cca-55974f2e12f0","Type":"ContainerDied","Data":"0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b"}
Jul 28 01:47:28.878141 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.877823    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l" event={"ID":"635abc6b-b5be-420a-9cca-55974f2e12f0","Type":"ContainerDied","Data":"9e0ccc2ac1b6d5f63fd7ab7d007b92818cd2181f01a598f2e5360ed25b0b3bec"}
Jul 28 01:47:28.878141 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.877863    2579 scope.go:117] "RemoveContainer" containerID="0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89"
Jul 28 01:47:28.878386 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.878360    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 01:47:28.888676 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.888657    2579 scope.go:117] "RemoveContainer" containerID="0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b"
Jul 28 01:47:28.895767 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.895749    2579 scope.go:117] "RemoveContainer" containerID="0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89"
Jul 28 01:47:28.896016 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:47:28.895999    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89\": container with ID starting with 0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89 not found: ID does not exist" containerID="0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89"
Jul 28 01:47:28.896072 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.896022    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89"} err="failed to get container status \"0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89\": rpc error: code = NotFound desc = could not find container \"0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89\": container with ID starting with 0e316ccfdfd9f7fcbad4bd6ac7e15bee1f18b0f20f6f64a6786869a8c04efd89 not found: ID does not exist"
Jul 28 01:47:28.896072 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.896038    2579 scope.go:117] "RemoveContainer" containerID="0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b"
Jul 28 01:47:28.896236 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:47:28.896221    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b\": container with ID starting with 0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b not found: ID does not exist" containerID="0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b"
Jul 28 01:47:28.896275 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.896239    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b"} err="failed to get container status \"0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b\": rpc error: code = NotFound desc = could not find container \"0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b\": container with ID starting with 0b82642d3e24243393a2dbe42dc3616dfec9cc902df48b9da0ba0353211db48b not found: ID does not exist"
Jul 28 01:47:28.902018 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.901992    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"]
Jul 28 01:47:28.906823 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:28.906798    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3d51c-predictor-56c57c964b-rtd7l"]
Jul 28 01:47:29.163760 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:29.163730    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" path="/var/lib/kubelet/pods/635abc6b-b5be-420a-9cca-55974f2e12f0/volumes"
Jul 28 01:47:33.737628 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:33.737589    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 01:47:33.882765 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:33.882738    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:47:33.883180 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:33.883150    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 01:47:43.738444 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:43.738415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:47:43.884125 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:43.884087    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 01:47:53.883322 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:47:53.883283    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 01:48:03.883587 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:03.883503    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 01:48:13.884636 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:13.884603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:48:15.876916 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.876882    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"]
Jul 28 01:48:15.877307 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.877127    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container" containerID="cri-o://c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3" gracePeriod=30
Jul 28 01:48:15.877307 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.877161    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kube-rbac-proxy" containerID="cri-o://3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d" gracePeriod=30
Jul 28 01:48:15.989973 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.989939    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"]
Jul 28 01:48:15.990312 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990299    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container"
Jul 28 01:48:15.990357 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990314    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container"
Jul 28 01:48:15.990357 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990329    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843"
Jul 28 01:48:15.990357 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990334    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843"
Jul 28 01:48:15.990357 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990345    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kube-rbac-proxy"
Jul 28 01:48:15.990357 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990351    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kube-rbac-proxy"
Jul 28 01:48:15.990524 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990403    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kube-rbac-proxy"
Jul 28 01:48:15.990524 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990413    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="635abc6b-b5be-420a-9cca-55974f2e12f0" containerName="kserve-container"
Jul 28 01:48:15.990524 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.990422    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d8ae9df-549f-41ea-9e81-7860ac4a4cb8" containerName="switch-graph-33843"
Jul 28 01:48:15.993130 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.993111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:15.995378 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.995354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5a4a0-predictor-serving-cert\""
Jul 28 01:48:15.995624 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:15.995610    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5a4a0-kube-rbac-proxy-sar-config\""
Jul 28 01:48:16.002632 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.002604    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"]
Jul 28 01:48:16.088213 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.088179    2579 generic.go:356] "Generic (PLEG): container finished" podID="8b436056-36ce-43e0-b251-495797c91744" containerID="3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d" exitCode=2
Jul 28 01:48:16.088358 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.088252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" event={"ID":"8b436056-36ce-43e0-b251-495797c91744","Type":"ContainerDied","Data":"3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d"}
Jul 28 01:48:16.141374 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.141287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.141374 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.141361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.141601 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.141407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxf4\" (UniqueName: \"kubernetes.io/projected/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-api-access-zrxf4\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.242520 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.242475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.242696 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.242612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.242696 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.242656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxf4\" (UniqueName: \"kubernetes.io/projected/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-api-access-zrxf4\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.242798 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:48:16.242752    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-serving-cert: secret "error-404-isvc-5a4a0-predictor-serving-cert" not found
Jul 28 01:48:16.242839 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:48:16.242830    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls podName:195c4c17-1b51-4793-acd9-867ed85d86e5 nodeName:}" failed. No retries permitted until 2026-07-28 01:48:16.742811015 +0000 UTC m=+1356.049299784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls") pod "error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" (UID: "195c4c17-1b51-4793-acd9-867ed85d86e5") : secret "error-404-isvc-5a4a0-predictor-serving-cert" not found
Jul 28 01:48:16.243126 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.243107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.252939 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.252914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxf4\" (UniqueName: \"kubernetes.io/projected/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-api-access-zrxf4\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.746885 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.746845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.749168 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.749146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls\") pod \"error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") " pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:16.904231 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:16.904198    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:17.040854 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:17.040824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"]
Jul 28 01:48:17.042639 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:48:17.042612    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195c4c17_1b51_4793_acd9_867ed85d86e5.slice/crio-d676474e9da8d38a78aaf3abbbe35acd8ffdf475ed53d7ec2886059cb2a1dd93 WatchSource:0}: Error finding container d676474e9da8d38a78aaf3abbbe35acd8ffdf475ed53d7ec2886059cb2a1dd93: Status 404 returned error can't find the container with id d676474e9da8d38a78aaf3abbbe35acd8ffdf475ed53d7ec2886059cb2a1dd93
Jul 28 01:48:17.093692 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:17.093665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" event={"ID":"195c4c17-1b51-4793-acd9-867ed85d86e5","Type":"ContainerStarted","Data":"d676474e9da8d38a78aaf3abbbe35acd8ffdf475ed53d7ec2886059cb2a1dd93"}
Jul 28 01:48:18.099624 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:18.099590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" event={"ID":"195c4c17-1b51-4793-acd9-867ed85d86e5","Type":"ContainerStarted","Data":"3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed"}
Jul 28 01:48:18.100112 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:18.099654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" event={"ID":"195c4c17-1b51-4793-acd9-867ed85d86e5","Type":"ContainerStarted","Data":"d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e"}
Jul 28 01:48:18.100112 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:18.099687    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:18.116716 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:18.116665    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podStartSLOduration=3.116649154 podStartE2EDuration="3.116649154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:18.116434365 +0000 UTC m=+1357.422923147" watchObservedRunningTime="2026-07-28 01:48:18.116649154 +0000 UTC m=+1357.423137933"
Jul 28 01:48:18.732574 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:18.732535    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 28 01:48:19.029119 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.029095    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:48:19.062402 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.062373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b436056-36ce-43e0-b251-495797c91744-kube-rbac-proxy-sar-config\") pod \"8b436056-36ce-43e0-b251-495797c91744\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") "
Jul 28 01:48:19.062555 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.062427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9tk8\" (UniqueName: \"kubernetes.io/projected/8b436056-36ce-43e0-b251-495797c91744-kube-api-access-v9tk8\") pod \"8b436056-36ce-43e0-b251-495797c91744\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") "
Jul 28 01:48:19.062555 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.062469    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b436056-36ce-43e0-b251-495797c91744-proxy-tls\") pod \"8b436056-36ce-43e0-b251-495797c91744\" (UID: \"8b436056-36ce-43e0-b251-495797c91744\") "
Jul 28 01:48:19.062832 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.062803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b436056-36ce-43e0-b251-495797c91744-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b436056-36ce-43e0-b251-495797c91744" (UID: "8b436056-36ce-43e0-b251-495797c91744"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:48:19.064500 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.064453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b436056-36ce-43e0-b251-495797c91744-kube-api-access-v9tk8" (OuterVolumeSpecName: "kube-api-access-v9tk8") pod "8b436056-36ce-43e0-b251-495797c91744" (UID: "8b436056-36ce-43e0-b251-495797c91744"). InnerVolumeSpecName "kube-api-access-v9tk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:48:19.064595 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.064527    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b436056-36ce-43e0-b251-495797c91744-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b436056-36ce-43e0-b251-495797c91744" (UID: "8b436056-36ce-43e0-b251-495797c91744"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:48:19.106609 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.106578    2579 generic.go:356] "Generic (PLEG): container finished" podID="8b436056-36ce-43e0-b251-495797c91744" containerID="c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3" exitCode=0
Jul 28 01:48:19.106971 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.106627    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"
Jul 28 01:48:19.106971 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.106668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" event={"ID":"8b436056-36ce-43e0-b251-495797c91744","Type":"ContainerDied","Data":"c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3"}
Jul 28 01:48:19.106971 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.106732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6" event={"ID":"8b436056-36ce-43e0-b251-495797c91744","Type":"ContainerDied","Data":"cf37e1f77d68abf5b0c8d1a9fb8243549d08ea9ada12281c261d9c1e77549b89"}
Jul 28 01:48:19.106971 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.106769    2579 scope.go:117] "RemoveContainer" containerID="3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d"
Jul 28 01:48:19.107364 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.107294    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:19.108352 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.108316    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 01:48:19.115923 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.115871    2579 scope.go:117] "RemoveContainer" containerID="c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3"
Jul 28 01:48:19.123300 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.123283    2579 scope.go:117] "RemoveContainer" containerID="3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d"
Jul 28 01:48:19.123597 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:48:19.123577    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d\": container with ID starting with 3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d not found: ID does not exist" containerID="3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d"
Jul 28 01:48:19.123655 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.123608    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d"} err="failed to get container status \"3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d\": rpc error: code = NotFound desc = could not find container \"3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d\": container with ID starting with 3301fc86e596cb22622cfc4f68abc15372221e5da92834670c92b79bf2010c4d not found: ID does not exist"
Jul 28 01:48:19.123655 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.123632    2579 scope.go:117] "RemoveContainer" containerID="c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3"
Jul 28 01:48:19.123862 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:48:19.123844    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3\": container with ID starting with c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3 not found: ID does not exist" containerID="c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3"
Jul 28 01:48:19.123915 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.123871    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3"} err="failed to get container status \"c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3\": rpc error: code = NotFound desc = could not find container \"c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3\": container with ID starting with c4c6ec1f27bcdd9144ca147b10682c8e3006cb96876af52ff6cf9b38d14761d3 not found: ID does not exist"
Jul 28 01:48:19.128940 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.128914    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"]
Jul 28 01:48:19.133459 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.133438    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-02c81-predictor-58cbbbb866-f6tf6"]
Jul 28 01:48:19.163234 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.163211    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b436056-36ce-43e0-b251-495797c91744" path="/var/lib/kubelet/pods/8b436056-36ce-43e0-b251-495797c91744/volumes"
Jul 28 01:48:19.163593 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.163565    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9tk8\" (UniqueName: \"kubernetes.io/projected/8b436056-36ce-43e0-b251-495797c91744-kube-api-access-v9tk8\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:48:19.163593 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.163589    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b436056-36ce-43e0-b251-495797c91744-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:48:19.163741 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:19.163605    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b436056-36ce-43e0-b251-495797c91744-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:48:20.112733 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:20.112687    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 01:48:25.117005 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:25.116975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:48:25.117460 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:25.117432    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 01:48:35.118228 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:35.118187    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 01:48:45.117931 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.117894    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 01:48:45.628349 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.628314    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"]
Jul 28 01:48:45.628608 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.628565    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container" containerID="cri-o://3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d" gracePeriod=30
Jul 28 01:48:45.628709 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.628597    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kube-rbac-proxy" containerID="cri-o://674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e" gracePeriod=30
Jul 28 01:48:45.758808 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.758775    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"]
Jul 28 01:48:45.759173 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.759160    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container"
Jul 28 01:48:45.759224 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.759175    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container"
Jul 28 01:48:45.759224 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.759184    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kube-rbac-proxy"
Jul 28 01:48:45.759224 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.759190    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kube-rbac-proxy"
Jul 28 01:48:45.759327 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.759241    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kube-rbac-proxy"
Jul 28 01:48:45.759327 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.759250    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b436056-36ce-43e0-b251-495797c91744" containerName="kserve-container"
Jul 28 01:48:45.762004 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.761989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.764870 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.764848    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-920f8-predictor-serving-cert\""
Jul 28 01:48:45.765011 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.764993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-920f8-kube-rbac-proxy-sar-config\""
Jul 28 01:48:45.777807 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.777783    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"]
Jul 28 01:48:45.885587 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.885499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.885763 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.885625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d3a5956-07f5-42d9-9961-606f2732abf6-proxy-tls\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.885763 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.885680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqq7\" (UniqueName: \"kubernetes.io/projected/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-api-access-mvqq7\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.986374 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.986335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d3a5956-07f5-42d9-9961-606f2732abf6-proxy-tls\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.986583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.986389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqq7\" (UniqueName: \"kubernetes.io/projected/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-api-access-mvqq7\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.986583 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.986437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.987104 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.987080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.988808 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.988788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d3a5956-07f5-42d9-9961-606f2732abf6-proxy-tls\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:45.995498 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:45.995464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqq7\" (UniqueName: \"kubernetes.io/projected/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-api-access-mvqq7\") pod \"error-404-isvc-920f8-predictor-674db79dcb-hgrcz\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") " pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:46.072795 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:46.072764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:46.206869 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:46.206838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"]
Jul 28 01:48:46.208612 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:48:46.208587    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3a5956_07f5_42d9_9961_606f2732abf6.slice/crio-50cc148ec030704da872e95da32823c4d160849e1c877770280436be3def12d8 WatchSource:0}: Error finding container 50cc148ec030704da872e95da32823c4d160849e1c877770280436be3def12d8: Status 404 returned error can't find the container with id 50cc148ec030704da872e95da32823c4d160849e1c877770280436be3def12d8
Jul 28 01:48:46.230827 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:46.230733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" event={"ID":"6d3a5956-07f5-42d9-9961-606f2732abf6","Type":"ContainerStarted","Data":"50cc148ec030704da872e95da32823c4d160849e1c877770280436be3def12d8"}
Jul 28 01:48:46.233352 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:46.233329    2579 generic.go:356] "Generic (PLEG): container finished" podID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerID="674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e" exitCode=2
Jul 28 01:48:46.233467 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:46.233387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" event={"ID":"c70de991-ec99-4f4a-a9af-44a84b89704b","Type":"ContainerDied","Data":"674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e"}
Jul 28 01:48:47.239111 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:47.239076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" event={"ID":"6d3a5956-07f5-42d9-9961-606f2732abf6","Type":"ContainerStarted","Data":"1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9"}
Jul 28 01:48:47.239602 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:47.239142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" event={"ID":"6d3a5956-07f5-42d9-9961-606f2732abf6","Type":"ContainerStarted","Data":"66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1"}
Jul 28 01:48:47.239602 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:47.239175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:47.239602 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:47.239205    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:47.240647 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:47.240622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 01:48:47.259565 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:47.259518    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podStartSLOduration=2.259503692 podStartE2EDuration="2.259503692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:47.258561863 +0000 UTC m=+1386.565050635" watchObservedRunningTime="2026-07-28 01:48:47.259503692 +0000 UTC m=+1386.565992469"
Jul 28 01:48:48.243736 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:48.243694    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 01:48:48.783762 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:48.783741    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:48:48.912558 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:48.912525    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-rbac-proxy-sar-config\") pod \"c70de991-ec99-4f4a-a9af-44a84b89704b\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") "
Jul 28 01:48:48.912717 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:48.912590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls\") pod \"c70de991-ec99-4f4a-a9af-44a84b89704b\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") "
Jul 28 01:48:48.912717 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:48.912625    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg8bq\" (UniqueName: \"kubernetes.io/projected/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-api-access-kg8bq\") pod \"c70de991-ec99-4f4a-a9af-44a84b89704b\" (UID: \"c70de991-ec99-4f4a-a9af-44a84b89704b\") "
Jul 28 01:48:48.912948 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:48.912920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c70de991-ec99-4f4a-a9af-44a84b89704b" (UID: "c70de991-ec99-4f4a-a9af-44a84b89704b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:48:48.914640 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:48.914613    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c70de991-ec99-4f4a-a9af-44a84b89704b" (UID: "c70de991-ec99-4f4a-a9af-44a84b89704b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:48:48.914750 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:48.914670    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-api-access-kg8bq" (OuterVolumeSpecName: "kube-api-access-kg8bq") pod "c70de991-ec99-4f4a-a9af-44a84b89704b" (UID: "c70de991-ec99-4f4a-a9af-44a84b89704b"). InnerVolumeSpecName "kube-api-access-kg8bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:48:49.014100 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.014069    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c70de991-ec99-4f4a-a9af-44a84b89704b-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:48:49.014100 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.014095    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg8bq\" (UniqueName: \"kubernetes.io/projected/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-api-access-kg8bq\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:48:49.014100 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.014104    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c70de991-ec99-4f4a-a9af-44a84b89704b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:48:49.249207 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.249176    2579 generic.go:356] "Generic (PLEG): container finished" podID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerID="3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d" exitCode=0
Jul 28 01:48:49.249603 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.249238    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"
Jul 28 01:48:49.249603 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.249250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" event={"ID":"c70de991-ec99-4f4a-a9af-44a84b89704b","Type":"ContainerDied","Data":"3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d"}
Jul 28 01:48:49.249603 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.249298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc" event={"ID":"c70de991-ec99-4f4a-a9af-44a84b89704b","Type":"ContainerDied","Data":"940dc72b47df69f86f96594786c24ecc9fa002d056ecbbc7716725b7d22a9c6f"}
Jul 28 01:48:49.249603 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.249326    2579 scope.go:117] "RemoveContainer" containerID="674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e"
Jul 28 01:48:49.257381 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.257364    2579 scope.go:117] "RemoveContainer" containerID="3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d"
Jul 28 01:48:49.264570 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.264554    2579 scope.go:117] "RemoveContainer" containerID="674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e"
Jul 28 01:48:49.264814 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:48:49.264796    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e\": container with ID starting with 674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e not found: ID does not exist" containerID="674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e"
Jul 28 01:48:49.264875 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.264821    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e"} err="failed to get container status \"674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e\": rpc error: code = NotFound desc = could not find container \"674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e\": container with ID starting with 674623e3991c66b6923a558ca2e23053dca228845a5b0abfc150da4c6ceca63e not found: ID does not exist"
Jul 28 01:48:49.264875 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.264836    2579 scope.go:117] "RemoveContainer" containerID="3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d"
Jul 28 01:48:49.265027 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:48:49.265013    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d\": container with ID starting with 3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d not found: ID does not exist" containerID="3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d"
Jul 28 01:48:49.265067 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.265032    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d"} err="failed to get container status \"3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d\": rpc error: code = NotFound desc = could not find container \"3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d\": container with ID starting with 3adc6234fd5412bb0d429dfb576f2ab5e6edee886657702e9b604bee6c1a1a7d not found: ID does not exist"
Jul 28 01:48:49.265178 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.265161    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"]
Jul 28 01:48:49.275856 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:49.275834    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2a946-predictor-85b9467f54-9t9hc"]
Jul 28 01:48:51.163458 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:51.163428    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" path="/var/lib/kubelet/pods/c70de991-ec99-4f4a-a9af-44a84b89704b/volumes"
Jul 28 01:48:53.248179 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:53.248151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:48:53.248687 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:53.248659    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 01:48:55.118431 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:48:55.118383    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 01:49:03.248764 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:49:03.248725    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 01:49:05.118328 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:49:05.118299    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:49:13.249070 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:49:13.249031    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 01:49:23.248636 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:49:23.248585    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 01:49:33.249946 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:49:33.249874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:50:41.155099 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:50:41.155072    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:50:41.155683 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:50:41.155288    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:55:41.184552 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:55:41.184527    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:55:41.185007 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:55:41.184596    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 01:57:40.871251 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.871213    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"]
Jul 28 01:57:40.873563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.871574    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kube-rbac-proxy"
Jul 28 01:57:40.873563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.871588    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kube-rbac-proxy"
Jul 28 01:57:40.873563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.871599    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container"
Jul 28 01:57:40.873563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.871604    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container"
Jul 28 01:57:40.873563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.871665    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kserve-container"
Jul 28 01:57:40.873563 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.871675    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70de991-ec99-4f4a-a9af-44a84b89704b" containerName="kube-rbac-proxy"
Jul 28 01:57:40.874572 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.874557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:40.876837 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.876817    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bc47b-predictor-serving-cert\""
Jul 28 01:57:40.876973 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.876956    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bc47b-kube-rbac-proxy-sar-config\""
Jul 28 01:57:40.889993 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.889970    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"]
Jul 28 01:57:40.935946 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.935918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdcd365-66da-4351-a436-7ad7647d133b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:40.936132 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.935963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:40.936132 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.936078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xh45\" (UniqueName: \"kubernetes.io/projected/7bdcd365-66da-4351-a436-7ad7647d133b-kube-api-access-2xh45\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:40.940804 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.940777    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"]
Jul 28 01:57:40.941032 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.940987    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container" containerID="cri-o://d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e" gracePeriod=30
Jul 28 01:57:40.941115 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:40.941021    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kube-rbac-proxy" containerID="cri-o://3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed" gracePeriod=30
Jul 28 01:57:41.037455 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.037414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xh45\" (UniqueName: \"kubernetes.io/projected/7bdcd365-66da-4351-a436-7ad7647d133b-kube-api-access-2xh45\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:41.037649 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.037494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdcd365-66da-4351-a436-7ad7647d133b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:41.037649 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.037556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:41.040222 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.040188    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bc47b-kube-rbac-proxy-sar-config\""
Jul 28 01:57:41.040577 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.040559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bc47b-predictor-serving-cert\""
Jul 28 01:57:41.047181 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.047158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xh45\" (UniqueName: \"kubernetes.io/projected/7bdcd365-66da-4351-a436-7ad7647d133b-kube-api-access-2xh45\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:41.047734 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:57:41.047716    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-serving-cert: secret "error-404-isvc-bc47b-predictor-serving-cert" not found
Jul 28 01:57:41.047817 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:57:41.047781    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls podName:7bdcd365-66da-4351-a436-7ad7647d133b nodeName:}" failed. No retries permitted until 2026-07-28 01:57:41.547761021 +0000 UTC m=+1920.854249787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls") pod "error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" (UID: "7bdcd365-66da-4351-a436-7ad7647d133b") : secret "error-404-isvc-bc47b-predictor-serving-cert" not found
Jul 28 01:57:41.048311 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.048286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdcd365-66da-4351-a436-7ad7647d133b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:41.542908 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.542874    2579 generic.go:356] "Generic (PLEG): container finished" podID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerID="3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed" exitCode=2
Jul 28 01:57:41.543097 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.542951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" event={"ID":"195c4c17-1b51-4793-acd9-867ed85d86e5","Type":"ContainerDied","Data":"3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed"}
Jul 28 01:57:41.641049 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.641013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:41.643554 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.643530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls\") pod \"error-404-isvc-bc47b-predictor-c6d4595c4-zrw66\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") " pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:41.785469 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.785423    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:41.918036 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:41.918000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"]
Jul 28 01:57:41.919834 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:57:41.919807    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdcd365_66da_4351_a436_7ad7647d133b.slice/crio-bfdba12170065f371e39999a238cb05091bf02850961654ce4f8ba7f2796aec7 WatchSource:0}: Error finding container bfdba12170065f371e39999a238cb05091bf02850961654ce4f8ba7f2796aec7: Status 404 returned error can't find the container with id bfdba12170065f371e39999a238cb05091bf02850961654ce4f8ba7f2796aec7
Jul 28 01:57:42.549500 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:42.549447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" event={"ID":"7bdcd365-66da-4351-a436-7ad7647d133b","Type":"ContainerStarted","Data":"7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983"}
Jul 28 01:57:42.549678 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:42.549527    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" event={"ID":"7bdcd365-66da-4351-a436-7ad7647d133b","Type":"ContainerStarted","Data":"99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632"}
Jul 28 01:57:42.549678 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:42.549549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" event={"ID":"7bdcd365-66da-4351-a436-7ad7647d133b","Type":"ContainerStarted","Data":"bfdba12170065f371e39999a238cb05091bf02850961654ce4f8ba7f2796aec7"}
Jul 28 01:57:42.549678 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:42.549572    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:42.549678 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:42.549593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:42.551119 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:42.551093    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 01:57:42.569161 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:42.569104    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podStartSLOduration=2.569089435 podStartE2EDuration="2.569089435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:57:42.568091548 +0000 UTC m=+1921.874580331" watchObservedRunningTime="2026-07-28 01:57:42.569089435 +0000 UTC m=+1921.875578215"
Jul 28 01:57:43.554320 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:43.554280    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 01:57:44.392541 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.392515    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:57:44.468086 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.468054    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls\") pod \"195c4c17-1b51-4793-acd9-867ed85d86e5\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") "
Jul 28 01:57:44.468240 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.468096    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-rbac-proxy-sar-config\") pod \"195c4c17-1b51-4793-acd9-867ed85d86e5\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") "
Jul 28 01:57:44.468240 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.468125    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zrxf4\" (UniqueName: \"kubernetes.io/projected/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-api-access-zrxf4\") pod \"195c4c17-1b51-4793-acd9-867ed85d86e5\" (UID: \"195c4c17-1b51-4793-acd9-867ed85d86e5\") "
Jul 28 01:57:44.468568 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.468543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "195c4c17-1b51-4793-acd9-867ed85d86e5" (UID: "195c4c17-1b51-4793-acd9-867ed85d86e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:57:44.470174 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.470150    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-api-access-zrxf4" (OuterVolumeSpecName: "kube-api-access-zrxf4") pod "195c4c17-1b51-4793-acd9-867ed85d86e5" (UID: "195c4c17-1b51-4793-acd9-867ed85d86e5"). InnerVolumeSpecName "kube-api-access-zrxf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:57:44.470263 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.470181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "195c4c17-1b51-4793-acd9-867ed85d86e5" (UID: "195c4c17-1b51-4793-acd9-867ed85d86e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:57:44.559773 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.559696    2579 generic.go:356] "Generic (PLEG): container finished" podID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerID="d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e" exitCode=0
Jul 28 01:57:44.559773 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.559741    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"
Jul 28 01:57:44.559773 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.559748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" event={"ID":"195c4c17-1b51-4793-acd9-867ed85d86e5","Type":"ContainerDied","Data":"d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e"}
Jul 28 01:57:44.560271 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.559795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf" event={"ID":"195c4c17-1b51-4793-acd9-867ed85d86e5","Type":"ContainerDied","Data":"d676474e9da8d38a78aaf3abbbe35acd8ffdf475ed53d7ec2886059cb2a1dd93"}
Jul 28 01:57:44.560271 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.559821    2579 scope.go:117] "RemoveContainer" containerID="3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed"
Jul 28 01:57:44.568259 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.568244    2579 scope.go:117] "RemoveContainer" containerID="d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e"
Jul 28 01:57:44.568790 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.568771    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/195c4c17-1b51-4793-acd9-867ed85d86e5-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:57:44.568880 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.568792    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:57:44.568880 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.568802    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxf4\" (UniqueName: \"kubernetes.io/projected/195c4c17-1b51-4793-acd9-867ed85d86e5-kube-api-access-zrxf4\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:57:44.575601 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.575573    2579 scope.go:117] "RemoveContainer" containerID="3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed"
Jul 28 01:57:44.575835 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:57:44.575814    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed\": container with ID starting with 3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed not found: ID does not exist" containerID="3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed"
Jul 28 01:57:44.575880 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.575844    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed"} err="failed to get container status \"3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed\": rpc error: code = NotFound desc = could not find container \"3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed\": container with ID starting with 3ed68e91d11ec66e2d6cc345e870ee0584c19a7afa440c3bd36aece0a801e0ed not found: ID does not exist"
Jul 28 01:57:44.575880 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.575861    2579 scope.go:117] "RemoveContainer" containerID="d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e"
Jul 28 01:57:44.576062 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:57:44.576045    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e\": container with ID starting with d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e not found: ID does not exist" containerID="d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e"
Jul 28 01:57:44.576098 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.576068    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e"} err="failed to get container status \"d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e\": rpc error: code = NotFound desc = could not find container \"d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e\": container with ID starting with d6d268723aec4504ab9e550ad9f70f8aac7ab3e095275cef5a9a4702ebe7c47e not found: ID does not exist"
Jul 28 01:57:44.580758 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.580738    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"]
Jul 28 01:57:44.587441 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:44.587421    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5a4a0-predictor-7b676c84f6-hmdgf"]
Jul 28 01:57:45.163479 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:45.163442    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" path="/var/lib/kubelet/pods/195c4c17-1b51-4793-acd9-867ed85d86e5/volumes"
Jul 28 01:57:48.558657 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:48.558629    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:57:48.559121 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:48.559092    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 01:57:58.559790 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:57:58.559754    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 01:58:08.559154 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:08.559115    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 01:58:10.347226 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.347190    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"]
Jul 28 01:58:10.347696 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.347465    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container" containerID="cri-o://66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1" gracePeriod=30
Jul 28 01:58:10.347696 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.347509    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kube-rbac-proxy" containerID="cri-o://1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9" gracePeriod=30
Jul 28 01:58:10.465122 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.465089    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"]
Jul 28 01:58:10.465500 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.465472    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container"
Jul 28 01:58:10.465559 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.465502    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container"
Jul 28 01:58:10.465559 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.465525    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kube-rbac-proxy"
Jul 28 01:58:10.465559 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.465532    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kube-rbac-proxy"
Jul 28 01:58:10.465654 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.465588    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kserve-container"
Jul 28 01:58:10.465654 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.465597    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="195c4c17-1b51-4793-acd9-867ed85d86e5" containerName="kube-rbac-proxy"
Jul 28 01:58:10.468508 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.468490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.470868 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.470843    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-704b9-predictor-serving-cert\""
Jul 28 01:58:10.470868 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.470844    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-704b9-kube-rbac-proxy-sar-config\""
Jul 28 01:58:10.481158 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.481136    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"]
Jul 28 01:58:10.586574 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.586540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30bf6ba-be23-4ecf-8f1c-65676592a689-proxy-tls\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.586730 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.586602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6x99\" (UniqueName: \"kubernetes.io/projected/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-api-access-d6x99\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.586730 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.586709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.676764 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.676724    2579 generic.go:356] "Generic (PLEG): container finished" podID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerID="1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9" exitCode=2
Jul 28 01:58:10.676936 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.676798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" event={"ID":"6d3a5956-07f5-42d9-9961-606f2732abf6","Type":"ContainerDied","Data":"1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9"}
Jul 28 01:58:10.687112 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.687088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.687203 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.687127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30bf6ba-be23-4ecf-8f1c-65676592a689-proxy-tls\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.687203 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.687172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6x99\" (UniqueName: \"kubernetes.io/projected/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-api-access-d6x99\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.687741 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.687724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.689539 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.689521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30bf6ba-be23-4ecf-8f1c-65676592a689-proxy-tls\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.695182 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.695160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6x99\" (UniqueName: \"kubernetes.io/projected/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-api-access-d6x99\") pod \"error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") " pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.778363 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.778329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:10.911777 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:10.911753    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"]
Jul 28 01:58:10.913340 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:58:10.913314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30bf6ba_be23_4ecf_8f1c_65676592a689.slice/crio-5b922a7edf85bc22fd2be0b31f156644b37467043bd6fa7ed7889c9f29ffc618 WatchSource:0}: Error finding container 5b922a7edf85bc22fd2be0b31f156644b37467043bd6fa7ed7889c9f29ffc618: Status 404 returned error can't find the container with id 5b922a7edf85bc22fd2be0b31f156644b37467043bd6fa7ed7889c9f29ffc618
Jul 28 01:58:11.683082 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:11.683039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" event={"ID":"c30bf6ba-be23-4ecf-8f1c-65676592a689","Type":"ContainerStarted","Data":"076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867"}
Jul 28 01:58:11.683507 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:11.683102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" event={"ID":"c30bf6ba-be23-4ecf-8f1c-65676592a689","Type":"ContainerStarted","Data":"330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1"}
Jul 28 01:58:11.683507 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:11.683127    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" event={"ID":"c30bf6ba-be23-4ecf-8f1c-65676592a689","Type":"ContainerStarted","Data":"5b922a7edf85bc22fd2be0b31f156644b37467043bd6fa7ed7889c9f29ffc618"}
Jul 28 01:58:11.683507 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:11.683154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:11.706167 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:11.706116    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podStartSLOduration=1.706100392 podStartE2EDuration="1.706100392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:58:11.703591145 +0000 UTC m=+1951.010079924" watchObservedRunningTime="2026-07-28 01:58:11.706100392 +0000 UTC m=+1951.012589172"
Jul 28 01:58:12.687014 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:12.686975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:12.688284 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:12.688260    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 01:58:13.244896 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.244859    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 28 01:58:13.249164 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.249141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 01:58:13.603084 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.603063    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:58:13.692730 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.692696    2579 generic.go:356] "Generic (PLEG): container finished" podID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerID="66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1" exitCode=0
Jul 28 01:58:13.693062 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.692747    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"
Jul 28 01:58:13.693062 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.692785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" event={"ID":"6d3a5956-07f5-42d9-9961-606f2732abf6","Type":"ContainerDied","Data":"66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1"}
Jul 28 01:58:13.693062 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.692845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz" event={"ID":"6d3a5956-07f5-42d9-9961-606f2732abf6","Type":"ContainerDied","Data":"50cc148ec030704da872e95da32823c4d160849e1c877770280436be3def12d8"}
Jul 28 01:58:13.693062 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.692871    2579 scope.go:117] "RemoveContainer" containerID="1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9"
Jul 28 01:58:13.693265 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.693182    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 01:58:13.700763 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.700748    2579 scope.go:117] "RemoveContainer" containerID="66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1"
Jul 28 01:58:13.707723 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.707706    2579 scope.go:117] "RemoveContainer" containerID="1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9"
Jul 28 01:58:13.707961 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:58:13.707941    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9\": container with ID starting with 1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9 not found: ID does not exist" containerID="1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9"
Jul 28 01:58:13.708042 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.707972    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9"} err="failed to get container status \"1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9\": rpc error: code = NotFound desc = could not find container \"1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9\": container with ID starting with 1f9f1dbb93fa3397022855763053203a66286a5cecc7f42f2b0e85075d6a00a9 not found: ID does not exist"
Jul 28 01:58:13.708042 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.707997    2579 scope.go:117] "RemoveContainer" containerID="66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1"
Jul 28 01:58:13.708217 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:58:13.708198    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1\": container with ID starting with 66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1 not found: ID does not exist" containerID="66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1"
Jul 28 01:58:13.708272 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.708224    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1"} err="failed to get container status \"66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1\": rpc error: code = NotFound desc = could not find container \"66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1\": container with ID starting with 66621f9d0eba793d3376e1f2c00fc3d9dd1f0126c921e238c9dd4adfb445a6c1 not found: ID does not exist"
Jul 28 01:58:13.713582 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.713562    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvqq7\" (UniqueName: \"kubernetes.io/projected/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-api-access-mvqq7\") pod \"6d3a5956-07f5-42d9-9961-606f2732abf6\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") "
Jul 28 01:58:13.713653 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.713616    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-rbac-proxy-sar-config\") pod \"6d3a5956-07f5-42d9-9961-606f2732abf6\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") "
Jul 28 01:58:13.713653 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.713640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d3a5956-07f5-42d9-9961-606f2732abf6-proxy-tls\") pod \"6d3a5956-07f5-42d9-9961-606f2732abf6\" (UID: \"6d3a5956-07f5-42d9-9961-606f2732abf6\") "
Jul 28 01:58:13.714025 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.713998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d3a5956-07f5-42d9-9961-606f2732abf6" (UID: "6d3a5956-07f5-42d9-9961-606f2732abf6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:58:13.715454 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.715429    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-api-access-mvqq7" (OuterVolumeSpecName: "kube-api-access-mvqq7") pod "6d3a5956-07f5-42d9-9961-606f2732abf6" (UID: "6d3a5956-07f5-42d9-9961-606f2732abf6"). InnerVolumeSpecName "kube-api-access-mvqq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:58:13.715598 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.715569    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d3a5956-07f5-42d9-9961-606f2732abf6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d3a5956-07f5-42d9-9961-606f2732abf6" (UID: "6d3a5956-07f5-42d9-9961-606f2732abf6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:58:13.814860 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.814783    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvqq7\" (UniqueName: \"kubernetes.io/projected/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-api-access-mvqq7\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:58:13.814860 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.814813    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d3a5956-07f5-42d9-9961-606f2732abf6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:58:13.815033 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:13.814898    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d3a5956-07f5-42d9-9961-606f2732abf6-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:58:14.014366 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:14.014336    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"]
Jul 28 01:58:14.020141 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:14.020119    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-920f8-predictor-674db79dcb-hgrcz"]
Jul 28 01:58:15.164064 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:15.164024    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" path="/var/lib/kubelet/pods/6d3a5956-07f5-42d9-9961-606f2732abf6/volumes"
Jul 28 01:58:18.559669 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:18.559634    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 28 01:58:18.698269 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:18.698242    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:18.698816 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:18.698791    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 01:58:28.559713 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:28.559685    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:58:28.699806 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:28.699764    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 01:58:38.698960 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:38.698879    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 01:58:48.698954 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:48.698912    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 01:58:50.953587 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:50.953553    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"]
Jul 28 01:58:50.954017 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:50.953780    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container" containerID="cri-o://99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632" gracePeriod=30
Jul 28 01:58:50.954369 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:50.954196    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kube-rbac-proxy" containerID="cri-o://7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983" gracePeriod=30
Jul 28 01:58:51.142849 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.142814    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"]
Jul 28 01:58:51.143186 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.143174    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kube-rbac-proxy"
Jul 28 01:58:51.143230 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.143188    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kube-rbac-proxy"
Jul 28 01:58:51.143230 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.143213    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container"
Jul 28 01:58:51.143230 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.143219    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container"
Jul 28 01:58:51.143323 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.143268    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kserve-container"
Jul 28 01:58:51.143323 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.143280    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3a5956-07f5-42d9-9961-606f2732abf6" containerName="kube-rbac-proxy"
Jul 28 01:58:51.146222 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.146202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.148900 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.148882    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7876a-kube-rbac-proxy-sar-config\""
Jul 28 01:58:51.148981 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.148954    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7876a-predictor-serving-cert\""
Jul 28 01:58:51.155956 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.155933    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"]
Jul 28 01:58:51.232967 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.232875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.233130 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.233000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8f296f4-a37f-475d-8b5a-cf834952cd3c-proxy-tls\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.233130 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.233046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5xfc\" (UniqueName: \"kubernetes.io/projected/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-api-access-f5xfc\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.334216 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.334176    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8f296f4-a37f-475d-8b5a-cf834952cd3c-proxy-tls\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.334458 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.334229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5xfc\" (UniqueName: \"kubernetes.io/projected/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-api-access-f5xfc\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.334458 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.334257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.335054 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.335023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.336778 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.336761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8f296f4-a37f-475d-8b5a-cf834952cd3c-proxy-tls\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.341995 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.341972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5xfc\" (UniqueName: \"kubernetes.io/projected/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-api-access-f5xfc\") pod \"error-404-isvc-7876a-predictor-7fd999b76-6ws2h\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") " pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.456901 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.456865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.585148 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.585035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"]
Jul 28 01:58:51.587419 ip-10-0-142-30 kubenswrapper[2579]: W0728 01:58:51.587392    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f296f4_a37f_475d_8b5a_cf834952cd3c.slice/crio-6f50d8b0a322a319e9ba4d60a2210eaa693de6a28c5d33fe23384d10aa2ade9a WatchSource:0}: Error finding container 6f50d8b0a322a319e9ba4d60a2210eaa693de6a28c5d33fe23384d10aa2ade9a: Status 404 returned error can't find the container with id 6f50d8b0a322a319e9ba4d60a2210eaa693de6a28c5d33fe23384d10aa2ade9a
Jul 28 01:58:51.860131 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.860073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" event={"ID":"c8f296f4-a37f-475d-8b5a-cf834952cd3c","Type":"ContainerStarted","Data":"9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4"}
Jul 28 01:58:51.860328 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.860224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:51.860328 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.860271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" event={"ID":"c8f296f4-a37f-475d-8b5a-cf834952cd3c","Type":"ContainerStarted","Data":"aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa"}
Jul 28 01:58:51.860328 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.860296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" event={"ID":"c8f296f4-a37f-475d-8b5a-cf834952cd3c","Type":"ContainerStarted","Data":"6f50d8b0a322a319e9ba4d60a2210eaa693de6a28c5d33fe23384d10aa2ade9a"}
Jul 28 01:58:51.861757 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.861735    2579 generic.go:356] "Generic (PLEG): container finished" podID="7bdcd365-66da-4351-a436-7ad7647d133b" containerID="7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983" exitCode=2
Jul 28 01:58:51.861858 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.861806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" event={"ID":"7bdcd365-66da-4351-a436-7ad7647d133b","Type":"ContainerDied","Data":"7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983"}
Jul 28 01:58:51.877620 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:51.877578    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podStartSLOduration=0.877566195 podStartE2EDuration="877.566195ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:58:51.875935719 +0000 UTC m=+1991.182424501" watchObservedRunningTime="2026-07-28 01:58:51.877566195 +0000 UTC m=+1991.184054975"
Jul 28 01:58:52.866055 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:52.866020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:52.867402 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:52.867375    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 01:58:53.554662 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:53.554619    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 28 01:58:53.870581 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:53.870541    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 01:58:54.214575 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.214552    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:58:54.257285 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.257253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xh45\" (UniqueName: \"kubernetes.io/projected/7bdcd365-66da-4351-a436-7ad7647d133b-kube-api-access-2xh45\") pod \"7bdcd365-66da-4351-a436-7ad7647d133b\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") "
Jul 28 01:58:54.257448 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.257341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdcd365-66da-4351-a436-7ad7647d133b-kube-rbac-proxy-sar-config\") pod \"7bdcd365-66da-4351-a436-7ad7647d133b\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") "
Jul 28 01:58:54.257448 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.257368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls\") pod \"7bdcd365-66da-4351-a436-7ad7647d133b\" (UID: \"7bdcd365-66da-4351-a436-7ad7647d133b\") "
Jul 28 01:58:54.257728 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.257704    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdcd365-66da-4351-a436-7ad7647d133b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bdcd365-66da-4351-a436-7ad7647d133b" (UID: "7bdcd365-66da-4351-a436-7ad7647d133b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:58:54.259452 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.259424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bdcd365-66da-4351-a436-7ad7647d133b" (UID: "7bdcd365-66da-4351-a436-7ad7647d133b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:58:54.259572 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.259553    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdcd365-66da-4351-a436-7ad7647d133b-kube-api-access-2xh45" (OuterVolumeSpecName: "kube-api-access-2xh45") pod "7bdcd365-66da-4351-a436-7ad7647d133b" (UID: "7bdcd365-66da-4351-a436-7ad7647d133b"). InnerVolumeSpecName "kube-api-access-2xh45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:58:54.358641 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.358598    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdcd365-66da-4351-a436-7ad7647d133b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:58:54.358641 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.358637    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdcd365-66da-4351-a436-7ad7647d133b-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:58:54.358814 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.358656    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xh45\" (UniqueName: \"kubernetes.io/projected/7bdcd365-66da-4351-a436-7ad7647d133b-kube-api-access-2xh45\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 01:58:54.875620 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.875580    2579 generic.go:356] "Generic (PLEG): container finished" podID="7bdcd365-66da-4351-a436-7ad7647d133b" containerID="99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632" exitCode=0
Jul 28 01:58:54.876040 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.875634    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"
Jul 28 01:58:54.876040 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.875662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" event={"ID":"7bdcd365-66da-4351-a436-7ad7647d133b","Type":"ContainerDied","Data":"99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632"}
Jul 28 01:58:54.876040 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.875725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66" event={"ID":"7bdcd365-66da-4351-a436-7ad7647d133b","Type":"ContainerDied","Data":"bfdba12170065f371e39999a238cb05091bf02850961654ce4f8ba7f2796aec7"}
Jul 28 01:58:54.876040 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.875752    2579 scope.go:117] "RemoveContainer" containerID="7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983"
Jul 28 01:58:54.884425 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.884408    2579 scope.go:117] "RemoveContainer" containerID="99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632"
Jul 28 01:58:54.891523 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.891470    2579 scope.go:117] "RemoveContainer" containerID="7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983"
Jul 28 01:58:54.891778 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:58:54.891745    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983\": container with ID starting with 7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983 not found: ID does not exist" containerID="7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983"
Jul 28 01:58:54.891830 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.891787    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983"} err="failed to get container status \"7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983\": rpc error: code = NotFound desc = could not find container \"7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983\": container with ID starting with 7f9ecbe6a6cf070cb710437f36e1f2d8cb4c2ac496097ce33e6dcc55cd582983 not found: ID does not exist"
Jul 28 01:58:54.891830 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.891805    2579 scope.go:117] "RemoveContainer" containerID="99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632"
Jul 28 01:58:54.892034 ip-10-0-142-30 kubenswrapper[2579]: E0728 01:58:54.892021    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632\": container with ID starting with 99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632 not found: ID does not exist" containerID="99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632"
Jul 28 01:58:54.892083 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.892036    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632"} err="failed to get container status \"99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632\": rpc error: code = NotFound desc = could not find container \"99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632\": container with ID starting with 99fd6f21e8c83b23633be071a378d65e0f189b4bb56304ac18a1cc577b596632 not found: ID does not exist"
Jul 28 01:58:54.897238 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.897217    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"]
Jul 28 01:58:54.902711 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:54.902689    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bc47b-predictor-c6d4595c4-zrw66"]
Jul 28 01:58:55.163617 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:55.163534    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" path="/var/lib/kubelet/pods/7bdcd365-66da-4351-a436-7ad7647d133b/volumes"
Jul 28 01:58:58.700590 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:58.700560    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 01:58:58.874919 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:58.874890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 01:58:58.875365 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:58:58.875340    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 01:59:08.875475 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:59:08.875434    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 01:59:18.876178 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:59:18.876140    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 01:59:28.875420 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:59:28.875382    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 01:59:38.876524 ip-10-0-142-30 kubenswrapper[2579]: I0728 01:59:38.876491    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 02:00:41.210406 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:00:41.210377    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:00:41.211552 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:00:41.211531    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:05:41.235854 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:05:41.235823    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:05:41.238332 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:05:41.237349    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:08:05.913138 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:05.913105    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"]
Jul 28 02:08:05.913705 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:05.913379    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kube-rbac-proxy" containerID="cri-o://9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4" gracePeriod=30
Jul 28 02:08:05.913705 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:05.913465    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container" containerID="cri-o://aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa" gracePeriod=30
Jul 28 02:08:06.286758 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:06.286673    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerID="9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4" exitCode=2
Jul 28 02:08:06.286758 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:06.286736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" event={"ID":"c8f296f4-a37f-475d-8b5a-cf834952cd3c","Type":"ContainerDied","Data":"9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4"}
Jul 28 02:08:08.871506 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:08.871443    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 02:08:08.875689 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:08.875663    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 02:08:09.062991 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.062967    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 02:08:09.129131 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.129046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5xfc\" (UniqueName: \"kubernetes.io/projected/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-api-access-f5xfc\") pod \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") "
Jul 28 02:08:09.129131 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.129088    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-rbac-proxy-sar-config\") pod \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") "
Jul 28 02:08:09.129341 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.129280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8f296f4-a37f-475d-8b5a-cf834952cd3c-proxy-tls\") pod \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\" (UID: \"c8f296f4-a37f-475d-8b5a-cf834952cd3c\") "
Jul 28 02:08:09.129397 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.129382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8f296f4-a37f-475d-8b5a-cf834952cd3c" (UID: "c8f296f4-a37f-475d-8b5a-cf834952cd3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:08:09.129568 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.129551    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 02:08:09.131099 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.131077    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f296f4-a37f-475d-8b5a-cf834952cd3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8f296f4-a37f-475d-8b5a-cf834952cd3c" (UID: "c8f296f4-a37f-475d-8b5a-cf834952cd3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:08:09.131191 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.131173    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-api-access-f5xfc" (OuterVolumeSpecName: "kube-api-access-f5xfc") pod "c8f296f4-a37f-475d-8b5a-cf834952cd3c" (UID: "c8f296f4-a37f-475d-8b5a-cf834952cd3c"). InnerVolumeSpecName "kube-api-access-f5xfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:08:09.230068 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.230038    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5xfc\" (UniqueName: \"kubernetes.io/projected/c8f296f4-a37f-475d-8b5a-cf834952cd3c-kube-api-access-f5xfc\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 02:08:09.230068 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.230066    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8f296f4-a37f-475d-8b5a-cf834952cd3c-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 02:08:09.301114 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.301081    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerID="aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa" exitCode=0
Jul 28 02:08:09.301261 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.301135    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"
Jul 28 02:08:09.301261 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.301162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" event={"ID":"c8f296f4-a37f-475d-8b5a-cf834952cd3c","Type":"ContainerDied","Data":"aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa"}
Jul 28 02:08:09.301261 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.301226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h" event={"ID":"c8f296f4-a37f-475d-8b5a-cf834952cd3c","Type":"ContainerDied","Data":"6f50d8b0a322a319e9ba4d60a2210eaa693de6a28c5d33fe23384d10aa2ade9a"}
Jul 28 02:08:09.301261 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.301254    2579 scope.go:117] "RemoveContainer" containerID="9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4"
Jul 28 02:08:09.309795 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.309780    2579 scope.go:117] "RemoveContainer" containerID="aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa"
Jul 28 02:08:09.317522 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.317504    2579 scope.go:117] "RemoveContainer" containerID="9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4"
Jul 28 02:08:09.317776 ip-10-0-142-30 kubenswrapper[2579]: E0728 02:08:09.317759    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4\": container with ID starting with 9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4 not found: ID does not exist" containerID="9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4"
Jul 28 02:08:09.317843 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.317784    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4"} err="failed to get container status \"9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4\": rpc error: code = NotFound desc = could not find container \"9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4\": container with ID starting with 9dd2fd6bc13a2af6100280714bf75f57d045565a1d94a7dfa3cb1c4943f6b7d4 not found: ID does not exist"
Jul 28 02:08:09.317843 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.317799    2579 scope.go:117] "RemoveContainer" containerID="aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa"
Jul 28 02:08:09.318015 ip-10-0-142-30 kubenswrapper[2579]: E0728 02:08:09.317998    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa\": container with ID starting with aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa not found: ID does not exist" containerID="aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa"
Jul 28 02:08:09.318054 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.318020    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa"} err="failed to get container status \"aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa\": rpc error: code = NotFound desc = could not find container \"aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa\": container with ID starting with aeea29a2f3ffcb27aa02fd8056b0939b174a26a8f46e3551215d15c62c2becfa not found: ID does not exist"
Jul 28 02:08:09.318377 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.318360    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"]
Jul 28 02:08:09.321365 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:09.321345    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7876a-predictor-7fd999b76-6ws2h"]
Jul 28 02:08:11.163763 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:08:11.163728    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" path="/var/lib/kubelet/pods/c8f296f4-a37f-475d-8b5a-cf834952cd3c/volumes"
Jul 28 02:10:41.261615 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:10:41.261502    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:10:41.264788 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:10:41.263415    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:15:39.746472 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:39.746441    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"]
Jul 28 02:15:39.747030 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:39.746689    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container" containerID="cri-o://330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1" gracePeriod=30
Jul 28 02:15:39.747030 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:39.746725    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kube-rbac-proxy" containerID="cri-o://076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867" gracePeriod=30
Jul 28 02:15:40.213836 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:40.213804    2579 generic.go:356] "Generic (PLEG): container finished" podID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerID="076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867" exitCode=2
Jul 28 02:15:40.214010 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:40.213885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" event={"ID":"c30bf6ba-be23-4ecf-8f1c-65676592a689","Type":"ContainerDied","Data":"076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867"}
Jul 28 02:15:41.287162 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:41.287055    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:15:41.295024 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:41.289318    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:15:42.988527 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:42.988504    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 02:15:43.149009 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.148966    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-rbac-proxy-sar-config\") pod \"c30bf6ba-be23-4ecf-8f1c-65676592a689\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") "
Jul 28 02:15:43.149197 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.149102    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30bf6ba-be23-4ecf-8f1c-65676592a689-proxy-tls\") pod \"c30bf6ba-be23-4ecf-8f1c-65676592a689\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") "
Jul 28 02:15:43.149197 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.149155    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6x99\" (UniqueName: \"kubernetes.io/projected/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-api-access-d6x99\") pod \"c30bf6ba-be23-4ecf-8f1c-65676592a689\" (UID: \"c30bf6ba-be23-4ecf-8f1c-65676592a689\") "
Jul 28 02:15:43.149341 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.149314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c30bf6ba-be23-4ecf-8f1c-65676592a689" (UID: "c30bf6ba-be23-4ecf-8f1c-65676592a689"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:15:43.151156 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.151126    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30bf6ba-be23-4ecf-8f1c-65676592a689-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c30bf6ba-be23-4ecf-8f1c-65676592a689" (UID: "c30bf6ba-be23-4ecf-8f1c-65676592a689"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:15:43.151277 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.151251    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-api-access-d6x99" (OuterVolumeSpecName: "kube-api-access-d6x99") pod "c30bf6ba-be23-4ecf-8f1c-65676592a689" (UID: "c30bf6ba-be23-4ecf-8f1c-65676592a689"). InnerVolumeSpecName "kube-api-access-d6x99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:15:43.228058 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.228023    2579 generic.go:356] "Generic (PLEG): container finished" podID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerID="330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1" exitCode=0
Jul 28 02:15:43.228215 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.228074    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"
Jul 28 02:15:43.228215 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.228105    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" event={"ID":"c30bf6ba-be23-4ecf-8f1c-65676592a689","Type":"ContainerDied","Data":"330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1"}
Jul 28 02:15:43.228215 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.228176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6" event={"ID":"c30bf6ba-be23-4ecf-8f1c-65676592a689","Type":"ContainerDied","Data":"5b922a7edf85bc22fd2be0b31f156644b37467043bd6fa7ed7889c9f29ffc618"}
Jul 28 02:15:43.228215 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.228211    2579 scope.go:117] "RemoveContainer" containerID="076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867"
Jul 28 02:15:43.236388 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.236373    2579 scope.go:117] "RemoveContainer" containerID="330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1"
Jul 28 02:15:43.243523 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.243499    2579 scope.go:117] "RemoveContainer" containerID="076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867"
Jul 28 02:15:43.243774 ip-10-0-142-30 kubenswrapper[2579]: E0728 02:15:43.243753    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867\": container with ID starting with 076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867 not found: ID does not exist" containerID="076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867"
Jul 28 02:15:43.243834 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.243781    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867"} err="failed to get container status \"076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867\": rpc error: code = NotFound desc = could not find container \"076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867\": container with ID starting with 076d16111fd39d2e7168ed510ccecbedbcca9abf587618ddb0e95ea1457fc867 not found: ID does not exist"
Jul 28 02:15:43.243834 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.243797    2579 scope.go:117] "RemoveContainer" containerID="330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1"
Jul 28 02:15:43.244055 ip-10-0-142-30 kubenswrapper[2579]: E0728 02:15:43.244035    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1\": container with ID starting with 330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1 not found: ID does not exist" containerID="330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1"
Jul 28 02:15:43.244204 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.244061    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1"} err="failed to get container status \"330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1\": rpc error: code = NotFound desc = could not find container \"330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1\": container with ID starting with 330e2ba19b3c52def38fc002bf7b2e9f3492490479eb6dd9abf147581e1b2bc1 not found: ID does not exist"
Jul 28 02:15:43.246174 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.246153    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"]
Jul 28 02:15:43.250541 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.250522    2579 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c30bf6ba-be23-4ecf-8f1c-65676592a689-proxy-tls\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 02:15:43.250640 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.250548    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6x99\" (UniqueName: \"kubernetes.io/projected/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-api-access-d6x99\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 02:15:43.250640 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.250563    2579 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c30bf6ba-be23-4ecf-8f1c-65676592a689-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-30.ec2.internal\" DevicePath \"\""
Jul 28 02:15:43.252206 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:43.252187    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-704b9-predictor-c58dcf5fd-fxxl6"]
Jul 28 02:15:45.163310 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:15:45.163279    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" path="/var/lib/kubelet/pods/c30bf6ba-be23-4ecf-8f1c-65676592a689/volumes"
Jul 28 02:16:13.368726 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:13.368698    2579 ???:1] "http: TLS handshake error from 10.0.134.158:51850: EOF"
Jul 28 02:16:13.376999 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:13.376976    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gd4x8_fa8e29a4-60fd-4509-ae9c-a6c9c7a1c021/global-pull-secret-syncer/0.log"
Jul 28 02:16:13.598766 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:13.598739    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jn5d4_63f51959-8dc3-4814-a455-b557cfedba90/konnectivity-agent/0.log"
Jul 28 02:16:13.673174 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:13.673101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-30.ec2.internal_d1c22b9cc929456a299125322077be27/haproxy/0.log"
Jul 28 02:16:17.275853 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.275818    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03dba6f2-b5b4-4a78-b505-44a084c41ff9/alertmanager/0.log"
Jul 28 02:16:17.311272 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.311239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03dba6f2-b5b4-4a78-b505-44a084c41ff9/config-reloader/0.log"
Jul 28 02:16:17.338810 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.338785    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03dba6f2-b5b4-4a78-b505-44a084c41ff9/kube-rbac-proxy-web/0.log"
Jul 28 02:16:17.366166 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.366138    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03dba6f2-b5b4-4a78-b505-44a084c41ff9/kube-rbac-proxy/0.log"
Jul 28 02:16:17.408872 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.408839    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03dba6f2-b5b4-4a78-b505-44a084c41ff9/kube-rbac-proxy-metric/0.log"
Jul 28 02:16:17.443870 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.443843    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03dba6f2-b5b4-4a78-b505-44a084c41ff9/prom-label-proxy/0.log"
Jul 28 02:16:17.480124 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.480100    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_03dba6f2-b5b4-4a78-b505-44a084c41ff9/init-config-reloader/0.log"
Jul 28 02:16:17.692234 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.692195    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-79c8557f5c-759hx_d19fecca-b180-4bbf-a4b4-dd38ffcd0338/metrics-server/0.log"
Jul 28 02:16:17.988367 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:17.988283    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndmc2_0852532d-b769-4dbd-a674-0e2472864fbc/node-exporter/0.log"
Jul 28 02:16:18.019057 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.019031    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndmc2_0852532d-b769-4dbd-a674-0e2472864fbc/kube-rbac-proxy/0.log"
Jul 28 02:16:18.054029 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.054005    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ndmc2_0852532d-b769-4dbd-a674-0e2472864fbc/init-textfile/0.log"
Jul 28 02:16:18.519335 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.519306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-9hcc4_e937ea66-9f7b-4193-af0d-b2a7f6fe4e54/prometheus-operator-admission-webhook/0.log"
Jul 28 02:16:18.648320 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.648295    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57bd896dcd-9zcrd_7fd27f23-16b3-4a2c-8e52-521bb3be8940/thanos-query/0.log"
Jul 28 02:16:18.671261 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.671236    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57bd896dcd-9zcrd_7fd27f23-16b3-4a2c-8e52-521bb3be8940/kube-rbac-proxy-web/0.log"
Jul 28 02:16:18.698101 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.698079    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57bd896dcd-9zcrd_7fd27f23-16b3-4a2c-8e52-521bb3be8940/kube-rbac-proxy/0.log"
Jul 28 02:16:18.725765 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.725739    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57bd896dcd-9zcrd_7fd27f23-16b3-4a2c-8e52-521bb3be8940/prom-label-proxy/0.log"
Jul 28 02:16:18.752842 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.752814    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57bd896dcd-9zcrd_7fd27f23-16b3-4a2c-8e52-521bb3be8940/kube-rbac-proxy-rules/0.log"
Jul 28 02:16:18.776720 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:18.776646    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57bd896dcd-9zcrd_7fd27f23-16b3-4a2c-8e52-521bb3be8940/kube-rbac-proxy-metrics/0.log"
Jul 28 02:16:21.440300 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:21.440273    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6dd55b56f5-zp9wm_be39a079-a272-4b93-be94-64e19d822653/console/0.log"
Jul 28 02:16:22.568579 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568546    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"]
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568854    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568865    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568873    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568879    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568889    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kube-rbac-proxy"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568895    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kube-rbac-proxy"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568910    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kube-rbac-proxy"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568915    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kube-rbac-proxy"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568924    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568929    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container"
Jul 28 02:16:22.568935 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568940    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kube-rbac-proxy"
Jul 28 02:16:22.569270 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568945    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kube-rbac-proxy"
Jul 28 02:16:22.569270 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568992    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kube-rbac-proxy"
Jul 28 02:16:22.569270 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.568999    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kube-rbac-proxy"
Jul 28 02:16:22.569270 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.569007    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f296f4-a37f-475d-8b5a-cf834952cd3c" containerName="kserve-container"
Jul 28 02:16:22.569270 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.569012    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bdcd365-66da-4351-a436-7ad7647d133b" containerName="kserve-container"
Jul 28 02:16:22.569270 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.569017    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kube-rbac-proxy"
Jul 28 02:16:22.569270 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.569025    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30bf6ba-be23-4ecf-8f1c-65676592a689" containerName="kserve-container"
Jul 28 02:16:22.571856 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.571838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.574130 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.574112    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8l4r\"/\"openshift-service-ca.crt\""
Jul 28 02:16:22.574886 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.574865    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p8l4r\"/\"default-dockercfg-gpg4w\""
Jul 28 02:16:22.574976 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.574905    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8l4r\"/\"kube-root-ca.crt\""
Jul 28 02:16:22.581806 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.581788    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"]
Jul 28 02:16:22.682968 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.682931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-proc\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.683155 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.682998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b728q\" (UniqueName: \"kubernetes.io/projected/4b5d1afa-543d-4428-98fc-83e48d89df80-kube-api-access-b728q\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.683155 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.683025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-lib-modules\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.683155 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.683051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-podres\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.683285 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.683140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-sys\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.783910 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.783877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-sys\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.784063 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.783934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-proc\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.784063 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.783970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b728q\" (UniqueName: \"kubernetes.io/projected/4b5d1afa-543d-4428-98fc-83e48d89df80-kube-api-access-b728q\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.784063 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.783988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-lib-modules\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.784063 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.784007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-podres\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.784063 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.784013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-sys\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.784255 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.784093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-proc\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.784255 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.784120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-podres\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.784255 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.784197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b5d1afa-543d-4428-98fc-83e48d89df80-lib-modules\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.791277 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.791260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b728q\" (UniqueName: \"kubernetes.io/projected/4b5d1afa-543d-4428-98fc-83e48d89df80-kube-api-access-b728q\") pod \"perf-node-gather-daemonset-hm22x\" (UID: \"4b5d1afa-543d-4428-98fc-83e48d89df80\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:22.881064 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:22.881041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:23.001638 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.001609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"]
Jul 28 02:16:23.004118 ip-10-0-142-30 kubenswrapper[2579]: W0728 02:16:23.004088    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b5d1afa_543d_4428_98fc_83e48d89df80.slice/crio-3f4a6bf7937e4b37af05883e0e890de48446a6d2edb2a7591bda80d570ee6aa3 WatchSource:0}: Error finding container 3f4a6bf7937e4b37af05883e0e890de48446a6d2edb2a7591bda80d570ee6aa3: Status 404 returned error can't find the container with id 3f4a6bf7937e4b37af05883e0e890de48446a6d2edb2a7591bda80d570ee6aa3
Jul 28 02:16:23.020269 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.020170    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pbrls_227a00b4-064f-4fef-af22-0a95f22a113c/dns/0.log"
Jul 28 02:16:23.043800 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.043780    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pbrls_227a00b4-064f-4fef-af22-0a95f22a113c/kube-rbac-proxy/0.log"
Jul 28 02:16:23.120031 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.120013    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mjjb_65669d60-5522-49e7-a490-1c2ac00830b1/dns-node-resolver/0.log"
Jul 28 02:16:23.399197 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.399167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x" event={"ID":"4b5d1afa-543d-4428-98fc-83e48d89df80","Type":"ContainerStarted","Data":"e63712f55b507a2c323cba57bc9d4457dff9785bb0fcce975c04db4db7cea2ed"}
Jul 28 02:16:23.399378 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.399214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x" event={"ID":"4b5d1afa-543d-4428-98fc-83e48d89df80","Type":"ContainerStarted","Data":"3f4a6bf7937e4b37af05883e0e890de48446a6d2edb2a7591bda80d570ee6aa3"}
Jul 28 02:16:23.399378 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.399300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:23.416096 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.416055    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x" podStartSLOduration=1.416041367 podStartE2EDuration="1.416041367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:16:23.414039773 +0000 UTC m=+3042.720528554" watchObservedRunningTime="2026-07-28 02:16:23.416041367 +0000 UTC m=+3042.722530177"
Jul 28 02:16:23.657032 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.656953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c6d86cc9-69rg5_a4498e9e-3eff-445b-b3af-23d0ad786c73/registry/0.log"
Jul 28 02:16:23.707672 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:23.707640    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kkv68_ffaa729e-9e90-4a5d-825c-d61951b1ca92/node-ca/0.log"
Jul 28 02:16:24.865979 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:24.865949    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4wn9f_e2fe9fdd-797b-4c19-91c2-91218d0bf2ca/serve-healthcheck-canary/0.log"
Jul 28 02:16:25.399403 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:25.399378    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m789s_6098fd8c-fce4-47e1-a5a1-bd9a75d1f937/kube-rbac-proxy/0.log"
Jul 28 02:16:25.421081 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:25.421060    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m789s_6098fd8c-fce4-47e1-a5a1-bd9a75d1f937/exporter/0.log"
Jul 28 02:16:25.443724 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:25.443700    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m789s_6098fd8c-fce4-47e1-a5a1-bd9a75d1f937/extractor/0.log"
Jul 28 02:16:27.778193 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:27.778129    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-ff96d5959-bvgb7_b96ed6ac-bb97-4df9-8612-ca316deec364/manager/0.log"
Jul 28 02:16:27.814222 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:27.814188    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-wlfjl_5e777faa-1be8-4281-b7c5-c0839a28b761/manager/0.log"
Jul 28 02:16:28.025064 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:28.025038    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-zm4ml_17899d4e-0142-4ca3-ad08-8ea958f54038/seaweedfs/0.log"
Jul 28 02:16:29.413331 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:29.413300    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-hm22x"
Jul 28 02:16:33.766301 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:33.766277    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6knn_db004963-7a39-48ea-be4f-286fada36b32/kube-multus-additional-cni-plugins/0.log"
Jul 28 02:16:33.789754 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:33.789733    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6knn_db004963-7a39-48ea-be4f-286fada36b32/egress-router-binary-copy/0.log"
Jul 28 02:16:33.811139 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:33.811118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6knn_db004963-7a39-48ea-be4f-286fada36b32/cni-plugins/0.log"
Jul 28 02:16:33.833123 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:33.833102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6knn_db004963-7a39-48ea-be4f-286fada36b32/bond-cni-plugin/0.log"
Jul 28 02:16:33.854874 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:33.854850    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6knn_db004963-7a39-48ea-be4f-286fada36b32/routeoverride-cni/0.log"
Jul 28 02:16:33.876660 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:33.876639    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6knn_db004963-7a39-48ea-be4f-286fada36b32/whereabouts-cni-bincopy/0.log"
Jul 28 02:16:33.897948 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:33.897926    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w6knn_db004963-7a39-48ea-be4f-286fada36b32/whereabouts-cni/0.log"
Jul 28 02:16:33.934081 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:33.934054    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-crcwz_a6596447-1335-4ce0-8d05-4f8ff15ea382/kube-multus/0.log"
Jul 28 02:16:34.131058 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:34.131018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xskmq_1c6f1795-9275-4b38-ba50-f2653f2b58dc/network-metrics-daemon/0.log"
Jul 28 02:16:34.153116 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:34.153094    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xskmq_1c6f1795-9275-4b38-ba50-f2653f2b58dc/kube-rbac-proxy/0.log"
Jul 28 02:16:35.113143 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.113102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-controller/0.log"
Jul 28 02:16:35.129950 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.129922    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/1.log"
Jul 28 02:16:35.130145 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.129964    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovn-acl-logging/0.log"
Jul 28 02:16:35.148297 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.148263    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/kube-rbac-proxy-node/0.log"
Jul 28 02:16:35.171524 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.171477    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 02:16:35.191977 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.191951    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/northd/0.log"
Jul 28 02:16:35.212690 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.212662    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/nbdb/0.log"
Jul 28 02:16:35.234427 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.234401    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/sbdb/0.log"
Jul 28 02:16:35.297811 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:35.297774    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbrmx_d0ad062e-f0ff-45cc-8a4a-db2e2a833a17/ovnkube-controller/0.log"
Jul 28 02:16:36.531496 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:36.531453    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2x9gm_f88f1a31-21c3-45c5-8cc3-f44b91806734/network-check-target-container/0.log"
Jul 28 02:16:37.526778 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:37.526754    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g4vkm_69084ae6-a530-4718-a6bc-2d9481b7c76c/iptables-alerter/0.log"
Jul 28 02:16:38.229775 ip-10-0-142-30 kubenswrapper[2579]: I0728 02:16:38.229727    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xp8sj_7a8c170e-1f5f-450c-8784-db3bef797f61/tuned/0.log"