Jul 09 01:59:29.997611 ip-10-0-131-83 systemd[1]: Starting Kubernetes Kubelet... Jul 09 01:59:30.377576 ip-10-0-131-83 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 01:59:30.377576 ip-10-0-131-83 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 01:59:30.377576 ip-10-0-131-83 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 01:59:30.377576 ip-10-0-131-83 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 01:59:30.377576 ip-10-0-131-83 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 01:59:30.378628 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.378502 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 01:59:30.386875 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386846 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 01:59:30.386875 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386870 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 01:59:30.386875 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386875 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 01:59:30.386875 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386878 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 01:59:30.386875 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386882 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 01:59:30.386875 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386885 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 01:59:30.386875 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386888 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386891 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386894 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386899 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386901 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386904 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386907 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386910 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386913 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386916 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386919 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386922 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386925 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386927 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386931 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386933 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386936 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386939 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386942 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386945 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 01:59:30.387175 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386948 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386951 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386954 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386957 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386960 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386963 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386966 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386969 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386971 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386974 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386976 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386979 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386983 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386986 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386990 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.386994 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387000 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387003 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387007 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 01:59:30.387713 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387010 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387013 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387016 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387019 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387031 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387036 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387039 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387042 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387045 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387047 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387050 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387053 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387056 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387059 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387063 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387065 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387068 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387071 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387074 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387077 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 01:59:30.388229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387079 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387082 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387085 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387088 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387090 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387094 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387098 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387100 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387103 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387106 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387109 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387112 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387115 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387118 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387120 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387123 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387126 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387129 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387134 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387136 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 01:59:30.388755 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387141 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387143 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387148 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387151 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387154 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387157 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387160 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387163 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387166 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387169 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387172 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387176 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387180 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.387183 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388025 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388034 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388037 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388041 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388044 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 01:59:30.389269 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388047 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388050 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388054 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388056 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388059 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388062 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388065 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388068 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388070 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388073 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388076 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388079 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388083 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388086 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388088 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388091 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388094 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388097 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388102 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 01:59:30.389817 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388106 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388109 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388112 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388115 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388118 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388123 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388127 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388130 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388133 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388136 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388139 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388142 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388146 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388149 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388152 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388155 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388158 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388161 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388165 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 01:59:30.390398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388169 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388172 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388176 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388179 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388182 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388185 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388187 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388191 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388194 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388197 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388200 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388202 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388205 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388207 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388210 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388213 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388216 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388219 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388221 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388224 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 01:59:30.390897 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388226 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388229 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388232 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388235 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388238 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388241 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388244 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388247 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388250 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388253 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388255 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388259 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388262 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388265 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388268 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388271 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388274 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388277 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388279 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388282 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 01:59:30.391444 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388285 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388289 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388292 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388295 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388298 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388300 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388303 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388306 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388309 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388312 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388315 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388317 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388337 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388340 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388343 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388346 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388839 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388845 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388848 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388851 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 01:59:30.391966 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388854 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388857 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388860 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388863 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388866 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388869 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388872 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388874 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388878 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388880 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388883 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388886 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388889 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388892 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388894 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388897 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388900 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388903 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388906 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388909 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 01:59:30.392511 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388912 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388915 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388918 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388920 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388923 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388926 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388928 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388931 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388934 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388938 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388942 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388945 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388948 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388951 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388955 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388957 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388960 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388963 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388965 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 01:59:30.393025 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388969 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388972 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388976 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388979 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388981 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388984 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388988 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388990 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388993 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388995 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.388998 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389000 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389005 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389009 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389012 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389016 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389019 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389022 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389025 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 01:59:30.393523 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389028 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389030 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389033 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389036 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389039 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389041 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389044 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389047 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389049 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389052 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389055 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389057 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389060 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389063 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389065 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389068 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389071 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389073 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389076 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389078 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 01:59:30.394013 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389084 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389087 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389090 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389093 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389096 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389099 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389104 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389107 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389110 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389118 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389121 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389123 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389126 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389129 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389132 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389135 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389138 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389211 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389219 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389226 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389230 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 01:59:30.394549 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389234 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389238 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389242 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389247 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389250 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389253 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389257 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389260 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389264 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389267 2578 flags.go:64] FLAG: --cgroup-root="" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389270 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389274 2578 flags.go:64] FLAG: --client-ca-file="" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389278 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389281 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389286 2578 flags.go:64] FLAG: --cluster-domain="" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389290 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389293 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389298 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389301 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389305 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389309 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389312 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389316 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389319 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 09 01:59:30.395079 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389339 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389343 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389346 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389349 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389354 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389357 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389360 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389363 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389366 2578 flags.go:64] FLAG: --enable-server="true" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389370 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389374 2578 flags.go:64] FLAG: --event-burst="100" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389377 2578 flags.go:64] FLAG: --event-qps="50" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389380 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389384 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389387 2578 flags.go:64] FLAG: --eviction-hard="" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389391 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389394 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389397 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389401 2578 flags.go:64] FLAG: --eviction-soft="" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389404 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389407 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389411 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389416 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389419 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389422 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 01:59:30.395725 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389427 2578 flags.go:64] FLAG: --feature-gates="" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389431 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389435 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389438 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389441 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389445 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389448 2578 flags.go:64] FLAG: --help="false" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389451 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389455 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389459 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389462 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389466 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389469 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389473 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389476 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389479 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389482 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389485 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389488 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389491 2578 flags.go:64] FLAG: --kube-reserved="" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389495 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389497 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389501 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389504 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 01:59:30.396448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389507 2578 flags.go:64] FLAG: --lock-file="" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389510 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389513 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389517 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389522 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389527 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389530 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389533 2578 flags.go:64] FLAG: --logging-format="text" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389539 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389542 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389545 2578 flags.go:64] FLAG: --manifest-url="" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389548 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389553 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389557 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389561 2578 flags.go:64] FLAG: --max-pods="110" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389564 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389568 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389571 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389574 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389577 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389580 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389583 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389591 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389594 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389598 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 01:59:30.397121 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389601 2578 flags.go:64] FLAG: --pod-cidr="" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389604 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389610 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389613 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389616 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389619 2578 flags.go:64] FLAG: --port="10250" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389622 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389626 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02dbecb79d037bd22" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389629 2578 flags.go:64] FLAG: --qos-reserved="" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389633 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389636 2578 flags.go:64] FLAG: --register-node="true" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389639 2578 flags.go:64] FLAG: --register-with-taints="" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389644 2578 flags.go:64] FLAG: --registry-burst="10" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389648 2578 flags.go:64] FLAG: --registry-qps="5" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389651 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389654 2578 flags.go:64] FLAG: --reserved-memory="" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389660 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389663 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389666 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389669 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389672 2578 flags.go:64] FLAG: --runonce="false" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389675 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389679 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389682 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389685 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 01:59:30.397813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389688 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389691 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389695 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389698 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389701 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389704 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389707 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389710 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389713 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389716 2578 flags.go:64] FLAG: --system-cgroups="" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389720 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389725 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389728 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389731 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389736 2578 flags.go:64] FLAG: --tls-min-version="" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389739 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389742 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389745 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389748 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389753 2578 flags.go:64] FLAG: --v="2" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389757 2578 flags.go:64] FLAG: --version="false" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389768 2578 flags.go:64] FLAG: --vmodule="" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389772 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.389778 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389925 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 01:59:30.398482 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389929 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389932 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389935 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389938 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389941 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389944 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389947 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389950 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389953 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389955 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389958 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389961 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389963 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389966 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389969 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389972 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389975 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389977 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389980 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 01:59:30.399134 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389983 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389985 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389988 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389991 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389993 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389996 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.389999 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390003 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390005 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390008 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390011 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390016 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390018 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390021 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390024 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390027 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390029 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390034 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390037 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390040 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 01:59:30.399647 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390043 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390045 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390048 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390051 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390054 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390057 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390060 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390063 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390065 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390068 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390071 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390073 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390076 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390079 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390082 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390084 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390087 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390092 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390095 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390100 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 01:59:30.400229 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390103 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390106 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390109 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390114 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390118 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390121 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390123 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390126 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390129 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390132 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390135 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390138 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390141 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390144 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390147 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390149 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390152 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390155 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390157 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390160 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 01:59:30.400862 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390163 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390166 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390168 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390171 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390173 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390176 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390179 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390182 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390186 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390189 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390192 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390197 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390200 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390203 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390206 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390210 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390213 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390216 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 01:59:30.401398 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.390219 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.390912 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.398078 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.398096 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.398894 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.398902 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398967 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398974 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398978 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398982 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398985 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398988 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398992 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398995 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.398998 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 01:59:30.401873 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399002 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399005 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399008 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399011 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399013 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399016 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399019 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399022 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399025 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399028 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399031 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399034 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399037 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399040 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399043 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399045 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399048 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399052 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399056 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 01:59:30.402316 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399060 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399065 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399069 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399072 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399075 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399079 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399084 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399087 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399090 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399093 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399095 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399098 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399101 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399104 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399107 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399111 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399113 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399116 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399119 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399121 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 01:59:30.402839 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399124 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399127 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399130 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399132 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399136 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399139 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399142 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399145 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399148 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399151 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399154 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399157 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399160 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399163 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399166 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399169 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399172 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399175 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399177 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399180 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 01:59:30.403381 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399183 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399186 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399189 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399192 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399194 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399197 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399200 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399203 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399206 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399209 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399211 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399214 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399218 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399222 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399225 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399229 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399233 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399236 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399239 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 01:59:30.403926 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399242 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399245 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399248 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399251 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399254 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399257 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399260 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399263 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399266 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399269 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399272 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399274 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.399279 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399458 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399466 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 01:59:30.404542 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399470 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399473 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399476 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399479 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399481 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399484 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399486 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399490 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399492 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399495 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399498 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399501 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399503 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399506 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399509 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399512 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399515 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399517 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399520 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399523 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 01:59:30.404943 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399525 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399528 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399531 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399535 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399538 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399541 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399544 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399547 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399550 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399553 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399555 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399558 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399561 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399564 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399566 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399569 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399571 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399574 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399577 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399580 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 01:59:30.405480 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399582 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399585 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399588 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399590 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399593 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399595 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399599 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399603 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399607 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399609 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399613 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399615 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399618 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399620 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399623 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399626 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399629 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399632 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399634 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399638 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 01:59:30.406119 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399640 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399644 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399647 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399649 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399653 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399656 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399659 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399662 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399665 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399668 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399670 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399675 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399678 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399681 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399685 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399688 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399690 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399694 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399697 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 01:59:30.406684 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399699 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399702 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399705 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399708 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399710 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399714 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399718 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399721 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399724 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399727 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399730 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399734 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399737 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399740 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399742 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399745 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399748 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 01:59:30.407174 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:30.399751 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 01:59:30.407623 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.399756 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 01:59:30.407623 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.399907 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 01:59:30.407623 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.402068 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 01:59:30.407623 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.403514 2578 server.go:1019] "Starting client certificate rotation" Jul 09 01:59:30.407623 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.403624 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 01:59:30.407623 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.403658 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 01:59:30.407623 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.406285 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 01:59:30.411297 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.411274 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 01:59:30.425502 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.425481 2578 log.go:25] "Validated CRI v1 runtime API" Jul 09 01:59:30.428019 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.427992 2578 log.go:25] "Validated CRI v1 image API" Jul 09 01:59:30.432519 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.432497 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 01:59:30.436838 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.436816 2578 fs.go:135] Filesystem UUIDs: map[231b35c9-022e-4871-85c8-2926a1171a7b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 874c78f2-6bcb-4be3-91ad-2bc29ee46ca4:/dev/nvme0n1p4] Jul 09 01:59:30.436944 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.436837 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 01:59:30.442160 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.442032 2578 manager.go:216] Machine: {Timestamp:2026-07-09 01:59:30.440828661 +0000 UTC m=+0.328991220 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091200 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec258ae8eee27efa77f6d7d1c32aed27 SystemUUID:ec258ae8-eee2-7efa-77f6-d7d1c32aed27 BootID:33a5066b-1a95-47b9-8ea1-f5c26e6a0d6b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6e:56:9c:f5:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6e:56:9c:f5:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:79:d4:4e:d3:59 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 01:59:30.442160 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.442151 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 01:59:30.442306 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.442240 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 01:59:30.443369 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.443318 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 01:59:30.443517 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.443372 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-83.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 01:59:30.444702 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.444690 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 01:59:30.444739 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.444704 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 01:59:30.444739 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.444713 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 01:59:30.445303 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.445292 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 01:59:30.446527 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.446513 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 01:59:30.446617 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.446588 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 01:59:30.449586 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.449574 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 09 01:59:30.449707 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.449697 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 01:59:30.449743 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.449729 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 09 01:59:30.449743 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.449740 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 01:59:30.449805 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.449751 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 01:59:30.449805 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.449761 2578 kubelet.go:398] "Adding apiserver pod source" Jul 09 01:59:30.449805 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.449772 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 01:59:30.450206 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.450191 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 01:59:30.450275 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.450212 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 01:59:30.454184 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.454167 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 01:59:30.456025 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.456006 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 01:59:30.456109 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.456039 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 01:59:30.457223 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457201 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 01:59:30.457376 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457364 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 01:59:30.457468 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457459 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 01:59:30.457537 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457529 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 01:59:30.457613 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457606 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 01:59:30.457684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457676 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 01:59:30.457756 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457748 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 01:59:30.457819 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457811 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 01:59:30.457887 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457879 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 01:59:30.457953 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.457945 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 01:59:30.458021 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.458014 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 01:59:30.458085 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.458078 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 01:59:30.459035 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.459018 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 01:59:30.459035 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.459032 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 01:59:30.462699 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.462674 2578 server.go:1268] "Started kubelet" Jul 09 01:59:30.462825 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.462772 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 01:59:30.462917 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.462831 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 01:59:30.462917 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.462913 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 01:59:30.463665 ip-10-0-131-83 systemd[1]: Started Kubernetes Kubelet. Jul 09 01:59:30.464750 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.464665 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 01:59:30.464909 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.464894 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 09 01:59:30.466529 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.466507 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-83.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 01:59:30.467313 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.467276 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 01:59:30.467313 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.467280 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 01:59:30.470940 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.470904 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 01:59:30.471484 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.471465 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 01:59:30.471592 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.471488 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 01:59:30.471838 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.471775 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:30.472087 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472065 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 01:59:30.472087 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472087 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 01:59:30.472222 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472175 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 01:59:30.472293 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472235 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 09 01:59:30.472293 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472245 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 09 01:59:30.472676 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472656 2578 factory.go:55] Registering systemd factory Jul 09 01:59:30.472676 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472677 2578 factory.go:223] Registration of the systemd container factory successfully Jul 09 01:59:30.472898 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472884 2578 factory.go:153] Registering CRI-O factory Jul 09 01:59:30.472898 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472897 2578 factory.go:223] Registration of the crio container factory successfully Jul 09 01:59:30.473009 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472941 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 01:59:30.473009 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472961 2578 factory.go:103] Registering Raw factory Jul 09 01:59:30.473009 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.472973 2578 manager.go:1196] Started watching for new ooms in manager Jul 09 01:59:30.474068 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.474052 2578 manager.go:349] Starting recovery of all containers Jul 09 01:59:30.477194 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.477169 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-83.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 01:59:30.477621 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.477596 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 01:59:30.478593 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.477427 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-83.ec2.internal.18c07c86b62d7c60 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-83.ec2.internal,UID:ip-10-0-131-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-83.ec2.internal,},FirstTimestamp:2026-07-09 01:59:30.462612576 +0000 UTC m=+0.350775134,LastTimestamp:2026-07-09 01:59:30.462612576 +0000 UTC m=+0.350775134,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-83.ec2.internal,}" Jul 09 01:59:30.487682 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.487664 2578 manager.go:354] Recovery completed Jul 09 01:59:30.491894 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.491879 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 01:59:30.494043 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.494026 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 09 01:59:30.494131 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.494062 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 01:59:30.494131 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.494078 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 09 01:59:30.494668 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.494652 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 01:59:30.494668 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.494662 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 01:59:30.494794 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.494665 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 01:59:30.494794 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.494700 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 09 01:59:30.495406 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.495389 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 09 01:59:30.495469 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.495420 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 01:59:30.495469 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.495434 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 09 01:59:30.496060 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.496047 2578 policy_none.go:49] "None policy: Start" Jul 09 01:59:30.496095 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.496065 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 01:59:30.496095 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.496076 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 01:59:30.497191 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.497177 2578 policy_none.go:47] "Start" Jul 09 01:59:30.497283 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.497154 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-83.ec2.internal.18c07c86b80d1614 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-83.ec2.internal,UID:ip-10-0-131-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-83.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-83.ec2.internal,},FirstTimestamp:2026-07-09 01:59:30.494043668 +0000 UTC m=+0.382206240,LastTimestamp:2026-07-09 01:59:30.494043668 +0000 UTC m=+0.382206240,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-83.ec2.internal,}" Jul 09 01:59:30.509767 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.509530 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-83.ec2.internal.18c07c86b80d79c4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-83.ec2.internal,UID:ip-10-0-131-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-83.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-83.ec2.internal,},FirstTimestamp:2026-07-09 01:59:30.494069188 +0000 UTC m=+0.382231747,LastTimestamp:2026-07-09 01:59:30.494069188 +0000 UTC m=+0.382231747,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-83.ec2.internal,}" Jul 09 01:59:30.518713 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.518635 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-83.ec2.internal.18c07c86b80db575 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-83.ec2.internal,UID:ip-10-0-131-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-83.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-83.ec2.internal,},FirstTimestamp:2026-07-09 01:59:30.494084469 +0000 UTC m=+0.382247029,LastTimestamp:2026-07-09 01:59:30.494084469 +0000 UTC m=+0.382247029,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-83.ec2.internal,}" Jul 09 01:59:30.532318 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.532214 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-83.ec2.internal.18c07c86b80d1614\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-83.ec2.internal.18c07c86b80d1614 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-83.ec2.internal,UID:ip-10-0-131-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-83.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-83.ec2.internal,},FirstTimestamp:2026-07-09 01:59:30.494043668 +0000 UTC m=+0.382206240,LastTimestamp:2026-07-09 01:59:30.495405532 +0000 UTC m=+0.383568092,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-83.ec2.internal,}" Jul 09 01:59:30.537019 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.537002 2578 manager.go:337] "Starting Device Plugin manager" Jul 09 01:59:30.537100 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.537041 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 01:59:30.537100 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.537054 2578 server.go:87] "Starting device plugin registration server" Jul 09 01:59:30.537374 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.537361 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 01:59:30.537411 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.537377 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 01:59:30.537526 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.537478 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 01:59:30.537704 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.537684 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 01:59:30.537704 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.537699 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 01:59:30.537860 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.537748 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r2dc2" Jul 09 01:59:30.538464 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.538439 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 01:59:30.538565 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.538498 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:30.544901 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.544822 2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-83.ec2.internal.18c07c86b80d79c4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-83.ec2.internal.18c07c86b80d79c4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-83.ec2.internal,UID:ip-10-0-131-83.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-83.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-83.ec2.internal,},FirstTimestamp:2026-07-09 01:59:30.494069188 +0000 UTC m=+0.382231747,LastTimestamp:2026-07-09 01:59:30.4954275 +0000 UTC m=+0.383590060,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-83.ec2.internal,}" Jul 09 01:59:30.547665 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.547645 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r2dc2" Jul 09 01:59:30.572406 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.572290 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 01:59:30.573732 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.573714 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 01:59:30.573825 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.573759 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 01:59:30.573825 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.573810 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 01:59:30.573916 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.573854 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 01:59:30.576842 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.576823 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 01:59:30.640393 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.640299 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 01:59:30.641471 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.641447 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 09 01:59:30.641591 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.641478 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 01:59:30.641591 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.641489 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 09 01:59:30.641591 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.641514 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.655000 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.654983 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.655082 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.655006 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-83.ec2.internal\": node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:30.674054 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.674027 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal"] Jul 09 01:59:30.674178 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.674087 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 01:59:30.675026 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.675011 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 09 01:59:30.675094 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.675040 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 01:59:30.675094 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.675054 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 09 01:59:30.676384 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.676371 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 01:59:30.676570 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.676557 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.676616 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.676607 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 01:59:30.677188 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.677170 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 09 01:59:30.677188 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.677185 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 09 01:59:30.677293 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.677202 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 01:59:30.677293 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.677216 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 09 01:59:30.677293 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.677203 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 01:59:30.677293 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.677289 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 09 01:59:30.678770 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.678754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.678840 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.678787 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 01:59:30.679511 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.679495 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 09 01:59:30.679573 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.679529 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 01:59:30.679573 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.679544 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 09 01:59:30.698690 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.698670 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:30.703093 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.703077 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-83.ec2.internal\" not found" node="ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.708957 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.708939 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-83.ec2.internal\" not found" node="ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.773981 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.773950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53be9f26ebbce9e8712878906521c6f8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-83.ec2.internal\" (UID: \"53be9f26ebbce9e8712878906521c6f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.773981 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.773983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4407acfac9b3a400518ad059d4a0580d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"4407acfac9b3a400518ad059d4a0580d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.774143 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.774003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4407acfac9b3a400518ad059d4a0580d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"4407acfac9b3a400518ad059d4a0580d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.799331 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.799297 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:30.875182 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.875139 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53be9f26ebbce9e8712878906521c6f8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-83.ec2.internal\" (UID: \"53be9f26ebbce9e8712878906521c6f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.875300 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.875189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4407acfac9b3a400518ad059d4a0580d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"4407acfac9b3a400518ad059d4a0580d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.875300 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.875209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4407acfac9b3a400518ad059d4a0580d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"4407acfac9b3a400518ad059d4a0580d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.875300 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.875231 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/53be9f26ebbce9e8712878906521c6f8-config\") pod \"kube-apiserver-proxy-ip-10-0-131-83.ec2.internal\" (UID: \"53be9f26ebbce9e8712878906521c6f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.875300 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.875248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4407acfac9b3a400518ad059d4a0580d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"4407acfac9b3a400518ad059d4a0580d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.875448 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:30.875315 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4407acfac9b3a400518ad059d4a0580d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal\" (UID: \"4407acfac9b3a400518ad059d4a0580d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:30.900817 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:30.900745 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.002153 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:31.002106 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.006567 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.006545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 09 01:59:31.013026 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.013003 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:31.102994 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:31.102944 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.204673 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:31.204581 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.305244 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:31.305208 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.349689 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.349667 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 01:59:31.404139 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.404088 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 01:59:31.404760 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.404257 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 01:59:31.404760 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.404314 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 01:59:31.406432 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:31.406414 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.507248 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:31.507154 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.550622 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.550563 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 01:54:31 +0000 UTC" deadline="2028-04-07 08:39:04.042128152 +0000 UTC" Jul 09 01:59:31.550776 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.550627 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15318h39m32.491506575s" Jul 09 01:59:31.607479 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:31.607446 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.645830 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.645801 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 01:59:31.647197 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.647172 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientMemory" Jul 09 01:59:31.647338 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.647214 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 01:59:31.647338 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.647230 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeHasSufficientPID" Jul 09 01:59:31.661384 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:31.661358 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53be9f26ebbce9e8712878906521c6f8.slice/crio-3a718e527e26ca3a1f7376056820aa04e39fca79c7f41ed50531d0bd41fd179c WatchSource:0}: Error finding container 3a718e527e26ca3a1f7376056820aa04e39fca79c7f41ed50531d0bd41fd179c: Status 404 returned error can't find the container with id 3a718e527e26ca3a1f7376056820aa04e39fca79c7f41ed50531d0bd41fd179c Jul 09 01:59:31.661820 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:31.661800 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4407acfac9b3a400518ad059d4a0580d.slice/crio-194bcdae29abdd777d74c4628422dc95259f44f4dcbe8f5d7500812137901420 WatchSource:0}: Error finding container 194bcdae29abdd777d74c4628422dc95259f44f4dcbe8f5d7500812137901420: Status 404 returned error can't find the container with id 194bcdae29abdd777d74c4628422dc95259f44f4dcbe8f5d7500812137901420 Jul 09 01:59:31.667964 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.667893 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 01:59:31.709155 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:31.709105 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-83.ec2.internal\" not found" Jul 09 01:59:31.719831 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.719697 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 01:59:31.726230 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.726199 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 01:59:31.773315 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.773211 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" Jul 09 01:59:31.792220 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.792198 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 01:59:31.793493 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.793479 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" Jul 09 01:59:31.803349 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:31.803329 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 01:59:32.224817 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.224738 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 01:59:32.453076 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.453043 2578 apiserver.go:52] "Watching apiserver" Jul 09 01:59:32.462869 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.462834 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 01:59:32.463491 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.463460 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-g2rkc","openshift-multus/network-metrics-daemon-t7q22","openshift-ovn-kubernetes/ovnkube-node-d7cp8","kube-system/konnectivity-agent-vkpgn","kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mq8t6","openshift-dns/node-resolver-hg8mp","openshift-image-registry/node-ca-w4w95","openshift-network-diagnostics/network-check-target-jzpk6","openshift-network-operator/iptables-alerter-zdk2r","kube-system/global-pull-secret-syncer-fg9k4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal","openshift-multus/multus-additional-cni-plugins-jqxdt"] Jul 09 01:59:32.465244 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.465214 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.467519 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.466695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.468753 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.468638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.469912 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.469891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:32.471933 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.471750 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 01:59:32.471933 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.471780 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q6s9f\"" Jul 09 01:59:32.471933 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.471848 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 01:59:32.472181 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.472092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.472547 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.472524 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 01:59:32.473042 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.473019 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zht4s\"" Jul 09 01:59:32.473200 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.473185 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.478916 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.479645 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.480050 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481051 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q4k4k\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481099 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481164 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fpc9q\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481345 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481457 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481653 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481705 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481777 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481898 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.481913 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.482208 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 01:59:32.482584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.482547 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 01:59:32.484010 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.483839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-host\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.484010 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.483882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70db81f0-b646-4ae3-ab59-02cb9bc52828-host\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.484010 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.483915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-os-release\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.486661 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.486627 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:32.486925 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.486900 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:32.487065 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.483953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-run\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.487464 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.487446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-node-log\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.487606 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.487592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-systemd-units\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.487732 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.487467 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 01:59:32.487822 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.487085 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 01:59:32.487993 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.487377 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.487720 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-cni-bin\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-ovnkube-script-lib\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-conf-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-multus-certs\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-kubernetes\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmhr\" (UniqueName: \"kubernetes.io/projected/7dfab8ba-2d59-47e7-abbe-b6695fb06715-kube-api-access-cmmhr\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-kubelet\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-etc-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488321 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488351 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ab7d58c-8222-4bb3-84b4-a09113c7fb8f-konnectivity-ca\") pod \"konnectivity-agent-vkpgn\" (UID: \"2ab7d58c-8222-4bb3-84b4-a09113c7fb8f\") " pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6dd0c8d-d57a-428f-9803-3c570479d35e-cni-binary-copy\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488398 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-hostroot\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-cni-bin\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-tuned\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f20951-58ef-4818-8715-c74ff5029e4d-tmp-dir\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ab7d58c-8222-4bb3-84b4-a09113c7fb8f-agent-certs\") pod \"konnectivity-agent-vkpgn\" (UID: \"2ab7d58c-8222-4bb3-84b4-a09113c7fb8f\") " pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:32.490064 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488529 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-var-lib-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-log-socket\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-ovnkube-config\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysconfig\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488627 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysctl-conf\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwmf\" (UniqueName: \"kubernetes.io/projected/e1f20951-58ef-4818-8715-c74ff5029e4d-kube-api-access-6qwmf\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-env-overrides\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-socket-dir-parent\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-etc-kubernetes\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488756 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj9x6\" (UniqueName: \"kubernetes.io/projected/a6dd0c8d-d57a-428f-9803-3c570479d35e-kube-api-access-mj9x6\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-modprobe-d\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-lib-modules\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1f20951-58ef-4818-8715-c74ff5029e4d-hosts-file\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488954 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1099b323-984e-47ae-9569-24c13d735e17-ovn-node-metrics-cert\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.488991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-cni-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489018 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70db81f0-b646-4ae3-ab59-02cb9bc52828-serviceca\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.490978 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-systemd\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-ovn\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-k8s-cni-cncf-io\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-cni-multus\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-kubelet\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489230 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-cni-netd\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-sys\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dfab8ba-2d59-47e7-abbe-b6695fb06715-tmp\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-system-cni-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-daemon-config\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvjq\" (UniqueName: \"kubernetes.io/projected/70db81f0-b646-4ae3-ab59-02cb9bc52828-kube-api-access-dwvjq\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489449 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-slash\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-run-netns\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-cnibin\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysctl-d\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.491824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489601 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-systemd\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-var-lib-kubelet\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjqwf\" (UniqueName: \"kubernetes.io/projected/1099b323-984e-47ae-9569-24c13d735e17-kube-api-access-zjqwf\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-netns\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489931 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tvzl4\"" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.489971 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xv9hb\"" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.490244 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.490336 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:32.492684 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.492106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.494205 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.493576 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 01:59:32.494205 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.493824 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 01:59:32.494205 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.493906 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8zsvj\"" Jul 09 01:59:32.494205 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.494099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.494205 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.494159 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:32.494700 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.494577 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.496289 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.495855 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 01:59:32.496289 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.495939 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h8qrt\"" Jul 09 01:59:32.496289 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.496136 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 01:59:32.497829 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.496720 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 01:59:32.497829 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.497392 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 01:59:32.497829 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.497672 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nrnww\"" Jul 09 01:59:32.498828 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.498802 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 01:59:32.498930 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.498806 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 01:59:32.506203 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.506180 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 01:59:32.537687 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.537606 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4t4kj" Jul 09 01:59:32.552617 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.552584 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4t4kj" Jul 09 01:59:32.574191 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.574164 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 01:59:32.580522 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.580464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" event={"ID":"53be9f26ebbce9e8712878906521c6f8","Type":"ContainerStarted","Data":"3a718e527e26ca3a1f7376056820aa04e39fca79c7f41ed50531d0bd41fd179c"} Jul 09 01:59:32.581822 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.581762 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" event={"ID":"4407acfac9b3a400518ad059d4a0580d","Type":"ContainerStarted","Data":"194bcdae29abdd777d74c4628422dc95259f44f4dcbe8f5d7500812137901420"} Jul 09 01:59:32.590469 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590432 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70db81f0-b646-4ae3-ab59-02cb9bc52828-serviceca\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.590629 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-systemd\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.590629 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-ovn\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.590629 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590571 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-ovn\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.590792 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590708 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-systemd\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.590792 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f27e32-08c7-4607-9805-e280bcbcd8fe-kubelet-config\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.590792 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cnibin\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.590948 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.590948 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-k8s-cni-cncf-io\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.590948 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-cni-multus\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591081 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590954 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-cni-multus\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591081 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-kubelet\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591081 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-cni-netd\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.591081 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-sys\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.591081 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-kubelet\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591081 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dfab8ba-2d59-47e7-abbe-b6695fb06715-tmp\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.591295 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f27e32-08c7-4607-9805-e280bcbcd8fe-dbus\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.591295 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-system-cni-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591295 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591124 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70db81f0-b646-4ae3-ab59-02cb9bc52828-serviceca\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.591295 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-daemon-config\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591295 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvjq\" (UniqueName: \"kubernetes.io/projected/70db81f0-b646-4ae3-ab59-02cb9bc52828-kube-api-access-dwvjq\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.591295 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591252 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-slash\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.591295 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.590981 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-k8s-cni-cncf-io\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591558 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591419 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 01:59:32.591558 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591429 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-sys\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.591558 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591479 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-system-cni-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591558 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591510 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-cni-netd\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.591558 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-run-netns\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.591730 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591565 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.591730 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-cnibin\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.591730 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.591730 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-os-release\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.591730 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-slash\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.591730 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591715 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysctl-d\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.591730 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591721 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-run-netns\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591741 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-systemd\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-run-ovn-kubernetes\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-var-lib-kubelet\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjqwf\" (UniqueName: \"kubernetes.io/projected/1099b323-984e-47ae-9569-24c13d735e17-kube-api-access-zjqwf\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591834 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-cnibin\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-netns\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-run-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-daemon-config\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-netns\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-host\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591913 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysctl-d\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70db81f0-b646-4ae3-ab59-02cb9bc52828-host\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-registration-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591919 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-var-lib-kubelet\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591992 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-host\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591997 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70db81f0-b646-4ae3-ab59-02cb9bc52828-host\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.591995 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-os-release\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.592056 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-systemd\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-run\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-node-log\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592138 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-os-release\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592166 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-run\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-systemd-units\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-node-log\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-cni-bin\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592298 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-cni-bin\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-systemd-units\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-ovnkube-script-lib\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-sys-fs\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592532 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592582 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ttlm\" (UniqueName: \"kubernetes.io/projected/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-kube-api-access-8ttlm\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.592954 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-conf-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-multus-certs\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-kubernetes\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmhr\" (UniqueName: \"kubernetes.io/projected/7dfab8ba-2d59-47e7-abbe-b6695fb06715-kube-api-access-cmmhr\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-kubelet\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-etc-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592897 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2wwb\" (UniqueName: \"kubernetes.io/projected/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-kube-api-access-l2wwb\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ab7d58c-8222-4bb3-84b4-a09113c7fb8f-konnectivity-ca\") pod \"konnectivity-agent-vkpgn\" (UID: \"2ab7d58c-8222-4bb3-84b4-a09113c7fb8f\") " pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6dd0c8d-d57a-428f-9803-3c570479d35e-cni-binary-copy\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-hostroot\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.592984 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-ovnkube-script-lib\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-host-slash\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-etc-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593070 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-cni-bin\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593117 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.593897 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593148 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-host-kubelet\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-tuned\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-kubernetes\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f20951-58ef-4818-8715-c74ff5029e4d-tmp-dir\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-iptables-alerter-script\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593349 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-var-lib-cni-bin\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2hx\" (UniqueName: \"kubernetes.io/projected/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-kube-api-access-mz2hx\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ab7d58c-8222-4bb3-84b4-a09113c7fb8f-agent-certs\") pod \"konnectivity-agent-vkpgn\" (UID: \"2ab7d58c-8222-4bb3-84b4-a09113c7fb8f\") " pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-var-lib-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-log-socket\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-ovnkube-config\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysconfig\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysctl-conf\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593672 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwmf\" (UniqueName: \"kubernetes.io/projected/e1f20951-58ef-4818-8715-c74ff5029e4d-kube-api-access-6qwmf\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6dd0c8d-d57a-428f-9803-3c570479d35e-cni-binary-copy\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.594757 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-env-overrides\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysconfig\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ab7d58c-8222-4bb3-84b4-a09113c7fb8f-konnectivity-ca\") pod \"konnectivity-agent-vkpgn\" (UID: \"2ab7d58c-8222-4bb3-84b4-a09113c7fb8f\") " pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-socket-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-conf-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znt8g\" (UniqueName: \"kubernetes.io/projected/c847f9eb-ebc3-40a4-a78c-e14cca47a814-kube-api-access-znt8g\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-socket-dir-parent\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-etc-kubernetes\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj9x6\" (UniqueName: \"kubernetes.io/projected/a6dd0c8d-d57a-428f-9803-3c570479d35e-kube-api-access-mj9x6\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.593975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-modprobe-d\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-lib-modules\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1f20951-58ef-4818-8715-c74ff5029e4d-hosts-file\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594141 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-socket-dir-parent\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594166 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-lib-modules\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-host-run-multus-certs\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-log-socket\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.595690 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594296 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-ovnkube-config\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594359 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-sysctl-conf\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-hostroot\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1099b323-984e-47ae-9569-24c13d735e17-ovn-node-metrics-cert\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594429 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-device-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594450 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1099b323-984e-47ae-9569-24c13d735e17-var-lib-openvswitch\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594465 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-cni-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594512 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1f20951-58ef-4818-8715-c74ff5029e4d-hosts-file\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f20951-58ef-4818-8715-c74ff5029e4d-tmp-dir\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594674 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-multus-cni-dir\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6dd0c8d-d57a-428f-9803-3c570479d35e-etc-kubernetes\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594881 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-modprobe-d\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.596572 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.594965 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1099b323-984e-47ae-9569-24c13d735e17-env-overrides\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.597253 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.597116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7dfab8ba-2d59-47e7-abbe-b6695fb06715-tmp\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.597397 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.597374 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7dfab8ba-2d59-47e7-abbe-b6695fb06715-etc-tuned\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.599234 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.599212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1099b323-984e-47ae-9569-24c13d735e17-ovn-node-metrics-cert\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.599550 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.599531 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ab7d58c-8222-4bb3-84b4-a09113c7fb8f-agent-certs\") pod \"konnectivity-agent-vkpgn\" (UID: \"2ab7d58c-8222-4bb3-84b4-a09113c7fb8f\") " pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:32.611016 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.607583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwmf\" (UniqueName: \"kubernetes.io/projected/e1f20951-58ef-4818-8715-c74ff5029e4d-kube-api-access-6qwmf\") pod \"node-resolver-hg8mp\" (UID: \"e1f20951-58ef-4818-8715-c74ff5029e4d\") " pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.611016 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.608225 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjqwf\" (UniqueName: \"kubernetes.io/projected/1099b323-984e-47ae-9569-24c13d735e17-kube-api-access-zjqwf\") pod \"ovnkube-node-d7cp8\" (UID: \"1099b323-984e-47ae-9569-24c13d735e17\") " pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.611016 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.608224 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj9x6\" (UniqueName: \"kubernetes.io/projected/a6dd0c8d-d57a-428f-9803-3c570479d35e-kube-api-access-mj9x6\") pod \"multus-g2rkc\" (UID: \"a6dd0c8d-d57a-428f-9803-3c570479d35e\") " pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.611016 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.608229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvjq\" (UniqueName: \"kubernetes.io/projected/70db81f0-b646-4ae3-ab59-02cb9bc52828-kube-api-access-dwvjq\") pod \"node-ca-w4w95\" (UID: \"70db81f0-b646-4ae3-ab59-02cb9bc52828\") " pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.612137 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.612026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmhr\" (UniqueName: \"kubernetes.io/projected/7dfab8ba-2d59-47e7-abbe-b6695fb06715-kube-api-access-cmmhr\") pod \"tuned-mq8t6\" (UID: \"7dfab8ba-2d59-47e7-abbe-b6695fb06715\") " pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.694830 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.694791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-registration-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.694830 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.694840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.694870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.694897 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-sys-fs\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.694923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.694948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.694949 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-registration-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.694975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ttlm\" (UniqueName: \"kubernetes.io/projected/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-kube-api-access-8ttlm\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695010 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2wwb\" (UniqueName: \"kubernetes.io/projected/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-kube-api-access-l2wwb\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-sys-fs\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-host-slash\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695102 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-iptables-alerter-script\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2hx\" (UniqueName: \"kubernetes.io/projected/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-kube-api-access-mz2hx\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-system-cni-dir\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-socket-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znt8g\" (UniqueName: \"kubernetes.io/projected/c847f9eb-ebc3-40a4-a78c-e14cca47a814-kube-api-access-znt8g\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-device-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f27e32-08c7-4607-9805-e280bcbcd8fe-kubelet-config\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cnibin\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695344 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695353 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695417 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f27e32-08c7-4607-9805-e280bcbcd8fe-dbus\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-os-release\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.695715 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695498 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-etc-selinux\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2f27e32-08c7-4607-9805-e280bcbcd8fe-dbus\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.695757 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.695858 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs podName:c847f9eb-ebc3-40a4-a78c-e14cca47a814 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:33.195805821 +0000 UTC m=+3.083968405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs") pod "network-metrics-daemon-t7q22" (UID: "c847f9eb-ebc3-40a4-a78c-e14cca47a814") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695919 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-host-slash\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.695983 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-device-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.696041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2f27e32-08c7-4607-9805-e280bcbcd8fe-kubelet-config\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.696055 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-iptables-alerter-script\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.696092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cnibin\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.696098 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-socket-dir\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.696144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.696155 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-os-release\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.696192 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:32.696513 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.696249 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret podName:c2f27e32-08c7-4607-9805-e280bcbcd8fe nodeName:}" failed. No retries permitted until 2026-07-09 01:59:33.196226386 +0000 UTC m=+3.084388952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret") pod "global-pull-secret-syncer-fg9k4" (UID: "c2f27e32-08c7-4607-9805-e280bcbcd8fe") : object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:32.697020 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.696592 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.697020 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.696624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.708213 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.708176 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 01:59:32.708213 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.708208 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 01:59:32.708213 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.708223 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lw9l7 for pod openshift-network-diagnostics/network-check-target-jzpk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:32.708532 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:32.708312 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7 podName:aaa28b10-cbbd-463c-a8a5-e5137a4070f6 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:33.208288814 +0000 UTC m=+3.096451383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lw9l7" (UniqueName: "kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7") pod "network-check-target-jzpk6" (UID: "aaa28b10-cbbd-463c-a8a5-e5137a4070f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:32.715754 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.715715 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znt8g\" (UniqueName: \"kubernetes.io/projected/c847f9eb-ebc3-40a4-a78c-e14cca47a814-kube-api-access-znt8g\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:32.715893 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.715822 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2hx\" (UniqueName: \"kubernetes.io/projected/1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3-kube-api-access-mz2hx\") pod \"iptables-alerter-zdk2r\" (UID: \"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3\") " pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.717117 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.717091 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ttlm\" (UniqueName: \"kubernetes.io/projected/3c624a3e-0848-4b58-bb45-a1d1dbb59b8a-kube-api-access-8ttlm\") pod \"multus-additional-cni-plugins-jqxdt\" (UID: \"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a\") " pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.718602 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.718551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2wwb\" (UniqueName: \"kubernetes.io/projected/59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd-kube-api-access-l2wwb\") pod \"aws-ebs-csi-driver-node-tqglh\" (UID: \"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.786584 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.786485 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hg8mp" Jul 09 01:59:32.797527 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.797491 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f20951_58ef_4818_8715_c74ff5029e4d.slice/crio-3e5275d9dcfbe35db9b0e34382de7c60d1f266e017143120b1da7c5ddf33601a WatchSource:0}: Error finding container 3e5275d9dcfbe35db9b0e34382de7c60d1f266e017143120b1da7c5ddf33601a: Status 404 returned error can't find the container with id 3e5275d9dcfbe35db9b0e34382de7c60d1f266e017143120b1da7c5ddf33601a Jul 09 01:59:32.799805 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.799778 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w4w95" Jul 09 01:59:32.809657 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.809622 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70db81f0_b646_4ae3_ab59_02cb9bc52828.slice/crio-4b00ad25833d16b624418cc5a719c12b6a9907500ff5cd482ffb6ba9ae4182c6 WatchSource:0}: Error finding container 4b00ad25833d16b624418cc5a719c12b6a9907500ff5cd482ffb6ba9ae4182c6: Status 404 returned error can't find the container with id 4b00ad25833d16b624418cc5a719c12b6a9907500ff5cd482ffb6ba9ae4182c6 Jul 09 01:59:32.812233 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.812207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:32.819310 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.819118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:32.822666 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.822635 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1099b323_984e_47ae_9569_24c13d735e17.slice/crio-99f280f26239a4e5cf8f928559ca5725a9d0063f8a7f192ccceef45d8ec37676 WatchSource:0}: Error finding container 99f280f26239a4e5cf8f928559ca5725a9d0063f8a7f192ccceef45d8ec37676: Status 404 returned error can't find the container with id 99f280f26239a4e5cf8f928559ca5725a9d0063f8a7f192ccceef45d8ec37676 Jul 09 01:59:32.828568 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.828545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g2rkc" Jul 09 01:59:32.832010 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.831964 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab7d58c_8222_4bb3_84b4_a09113c7fb8f.slice/crio-439f355d39ebe889f647268f93dda71e5a36c9c6c3dd94a4689f733fcea9f6cd WatchSource:0}: Error finding container 439f355d39ebe889f647268f93dda71e5a36c9c6c3dd94a4689f733fcea9f6cd: Status 404 returned error can't find the container with id 439f355d39ebe889f647268f93dda71e5a36c9c6c3dd94a4689f733fcea9f6cd Jul 09 01:59:32.837820 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.837793 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" Jul 09 01:59:32.840225 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.840195 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dd0c8d_d57a_428f_9803_3c570479d35e.slice/crio-7fcfe3af9da9ba359d6aeff8cb1113725877010d6bf9e7c391378e39b99c0dc6 WatchSource:0}: Error finding container 7fcfe3af9da9ba359d6aeff8cb1113725877010d6bf9e7c391378e39b99c0dc6: Status 404 returned error can't find the container with id 7fcfe3af9da9ba359d6aeff8cb1113725877010d6bf9e7c391378e39b99c0dc6 Jul 09 01:59:32.847026 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.846997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" Jul 09 01:59:32.850160 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.850127 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfab8ba_2d59_47e7_abbe_b6695fb06715.slice/crio-e4955ffd972b771abe9ccfcf0e5e71f06574555e8951610232e618115fadae16 WatchSource:0}: Error finding container e4955ffd972b771abe9ccfcf0e5e71f06574555e8951610232e618115fadae16: Status 404 returned error can't find the container with id e4955ffd972b771abe9ccfcf0e5e71f06574555e8951610232e618115fadae16 Jul 09 01:59:32.858124 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.858097 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c624a3e_0848_4b58_bb45_a1d1dbb59b8a.slice/crio-2a87a243d1bba03bb9c2e391fcd8810837ab250919f14724434c1e5ac67467ab WatchSource:0}: Error finding container 2a87a243d1bba03bb9c2e391fcd8810837ab250919f14724434c1e5ac67467ab: Status 404 returned error can't find the container with id 2a87a243d1bba03bb9c2e391fcd8810837ab250919f14724434c1e5ac67467ab Jul 09 01:59:32.865668 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.865573 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdk2r" Jul 09 01:59:32.871514 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:32.871459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" Jul 09 01:59:32.876012 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.875988 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b17fef7_10c2_4ebd_9f80_0f3c7417b0d3.slice/crio-debf6ea10ea892cd22f21bfddff501821b8a69b7cb8651162853c58973102ee9 WatchSource:0}: Error finding container debf6ea10ea892cd22f21bfddff501821b8a69b7cb8651162853c58973102ee9: Status 404 returned error can't find the container with id debf6ea10ea892cd22f21bfddff501821b8a69b7cb8651162853c58973102ee9 Jul 09 01:59:32.880860 ip-10-0-131-83 kubenswrapper[2578]: W0709 01:59:32.880835 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ecb41d_4d2b_4e4b_9d31_2d4b668dbfdd.slice/crio-74f19794e4018a7a01fbf515c451a7cadae432f8ba577bf246caa5702cb420e4 WatchSource:0}: Error finding container 74f19794e4018a7a01fbf515c451a7cadae432f8ba577bf246caa5702cb420e4: Status 404 returned error can't find the container with id 74f19794e4018a7a01fbf515c451a7cadae432f8ba577bf246caa5702cb420e4 Jul 09 01:59:33.200015 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.199931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:33.200015 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.199975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:33.200249 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.200104 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:33.200249 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.200107 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:33.200249 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.200176 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs podName:c847f9eb-ebc3-40a4-a78c-e14cca47a814 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:34.200156505 +0000 UTC m=+4.088319078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs") pod "network-metrics-daemon-t7q22" (UID: "c847f9eb-ebc3-40a4-a78c-e14cca47a814") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:33.200249 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.200196 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret podName:c2f27e32-08c7-4607-9805-e280bcbcd8fe nodeName:}" failed. No retries permitted until 2026-07-09 01:59:34.200187114 +0000 UTC m=+4.088349661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret") pod "global-pull-secret-syncer-fg9k4" (UID: "c2f27e32-08c7-4607-9805-e280bcbcd8fe") : object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:33.300523 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.300465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:33.300724 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.300675 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 01:59:33.300724 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.300699 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 01:59:33.300724 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.300714 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lw9l7 for pod openshift-network-diagnostics/network-check-target-jzpk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:33.300899 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.300778 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7 podName:aaa28b10-cbbd-463c-a8a5-e5137a4070f6 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:34.30076121 +0000 UTC m=+4.188923754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw9l7" (UniqueName: "kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7") pod "network-check-target-jzpk6" (UID: "aaa28b10-cbbd-463c-a8a5-e5137a4070f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:33.555442 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.555341 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 01:54:33 +0000 UTC" deadline="2028-01-26 05:36:32.7105932 +0000 UTC" Jul 09 01:59:33.555442 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.555395 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13587h36m59.155201111s" Jul 09 01:59:33.574920 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.574884 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:33.575100 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:33.575024 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:33.584813 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.584777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" event={"ID":"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd","Type":"ContainerStarted","Data":"74f19794e4018a7a01fbf515c451a7cadae432f8ba577bf246caa5702cb420e4"} Jul 09 01:59:33.586011 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.585982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" event={"ID":"7dfab8ba-2d59-47e7-abbe-b6695fb06715","Type":"ContainerStarted","Data":"e4955ffd972b771abe9ccfcf0e5e71f06574555e8951610232e618115fadae16"} Jul 09 01:59:33.587291 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.587247 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g2rkc" event={"ID":"a6dd0c8d-d57a-428f-9803-3c570479d35e","Type":"ContainerStarted","Data":"7fcfe3af9da9ba359d6aeff8cb1113725877010d6bf9e7c391378e39b99c0dc6"} Jul 09 01:59:33.588478 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.588453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkpgn" event={"ID":"2ab7d58c-8222-4bb3-84b4-a09113c7fb8f","Type":"ContainerStarted","Data":"439f355d39ebe889f647268f93dda71e5a36c9c6c3dd94a4689f733fcea9f6cd"} Jul 09 01:59:33.589768 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.589736 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdk2r" event={"ID":"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3","Type":"ContainerStarted","Data":"debf6ea10ea892cd22f21bfddff501821b8a69b7cb8651162853c58973102ee9"} Jul 09 01:59:33.591025 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.591003 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerStarted","Data":"2a87a243d1bba03bb9c2e391fcd8810837ab250919f14724434c1e5ac67467ab"} Jul 09 01:59:33.592312 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.592271 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"99f280f26239a4e5cf8f928559ca5725a9d0063f8a7f192ccceef45d8ec37676"} Jul 09 01:59:33.593553 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.593526 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w4w95" event={"ID":"70db81f0-b646-4ae3-ab59-02cb9bc52828","Type":"ContainerStarted","Data":"4b00ad25833d16b624418cc5a719c12b6a9907500ff5cd482ffb6ba9ae4182c6"} Jul 09 01:59:33.594732 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:33.594708 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hg8mp" event={"ID":"e1f20951-58ef-4818-8715-c74ff5029e4d","Type":"ContainerStarted","Data":"3e5275d9dcfbe35db9b0e34382de7c60d1f266e017143120b1da7c5ddf33601a"} Jul 09 01:59:34.211528 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:34.210611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:34.211528 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:34.210674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:34.211528 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.210852 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:34.211528 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.210921 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs podName:c847f9eb-ebc3-40a4-a78c-e14cca47a814 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:36.210900323 +0000 UTC m=+6.099062874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs") pod "network-metrics-daemon-t7q22" (UID: "c847f9eb-ebc3-40a4-a78c-e14cca47a814") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:34.211528 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.211366 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:34.211528 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.211422 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret podName:c2f27e32-08c7-4607-9805-e280bcbcd8fe nodeName:}" failed. No retries permitted until 2026-07-09 01:59:36.211403864 +0000 UTC m=+6.099566425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret") pod "global-pull-secret-syncer-fg9k4" (UID: "c2f27e32-08c7-4607-9805-e280bcbcd8fe") : object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:34.312218 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:34.311488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:34.312218 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.311734 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 01:59:34.312218 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.311763 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 01:59:34.312218 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.311777 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lw9l7 for pod openshift-network-diagnostics/network-check-target-jzpk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:34.312218 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.311844 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7 podName:aaa28b10-cbbd-463c-a8a5-e5137a4070f6 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:36.311823237 +0000 UTC m=+6.199985788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw9l7" (UniqueName: "kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7") pod "network-check-target-jzpk6" (UID: "aaa28b10-cbbd-463c-a8a5-e5137a4070f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:34.334308 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:34.333918 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 01:59:34.560989 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:34.556495 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 01:54:33 +0000 UTC" deadline="2028-03-07 15:29:07.498017563 +0000 UTC" Jul 09 01:59:34.560989 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:34.556544 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14581h29m32.94147891s" Jul 09 01:59:34.575284 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:34.575240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:34.575498 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:34.575301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:34.575498 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.575391 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:34.575884 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:34.575855 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:35.574998 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:35.574747 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:35.575506 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:35.575131 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:35.616649 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:35.616000 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" event={"ID":"4407acfac9b3a400518ad059d4a0580d","Type":"ContainerStarted","Data":"d1a54e447fc635a4e06b8458d86b07373fd4465490d977dafdc3f217d3e43f89"} Jul 09 01:59:35.637371 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:35.636682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" event={"ID":"53be9f26ebbce9e8712878906521c6f8","Type":"ContainerStarted","Data":"dff2f8fdaeedcf3fd99b56268f8718262e2c3030c25b8096a9741e715770f393"} Jul 09 01:59:36.228627 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:36.228583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:36.228824 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:36.228688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:36.228920 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.228826 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:36.228920 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.228893 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret podName:c2f27e32-08c7-4607-9805-e280bcbcd8fe nodeName:}" failed. No retries permitted until 2026-07-09 01:59:40.228875017 +0000 UTC m=+10.117037569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret") pod "global-pull-secret-syncer-fg9k4" (UID: "c2f27e32-08c7-4607-9805-e280bcbcd8fe") : object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:36.229363 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.229345 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:36.229450 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.229399 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs podName:c847f9eb-ebc3-40a4-a78c-e14cca47a814 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:40.229383399 +0000 UTC m=+10.117545953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs") pod "network-metrics-daemon-t7q22" (UID: "c847f9eb-ebc3-40a4-a78c-e14cca47a814") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:36.329607 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:36.329566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:36.329775 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.329748 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 01:59:36.329775 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.329768 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 01:59:36.329843 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.329781 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lw9l7 for pod openshift-network-diagnostics/network-check-target-jzpk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:36.329875 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.329846 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7 podName:aaa28b10-cbbd-463c-a8a5-e5137a4070f6 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:40.329827773 +0000 UTC m=+10.217990324 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw9l7" (UniqueName: "kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7") pod "network-check-target-jzpk6" (UID: "aaa28b10-cbbd-463c-a8a5-e5137a4070f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:36.576781 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:36.576683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:36.577262 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.576825 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:36.577358 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:36.577267 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:36.577412 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:36.577393 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:37.574696 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:37.574609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:37.574882 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:37.574770 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:37.642589 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:37.642557 2578 generic.go:356] "Generic (PLEG): container finished" podID="4407acfac9b3a400518ad059d4a0580d" containerID="d1a54e447fc635a4e06b8458d86b07373fd4465490d977dafdc3f217d3e43f89" exitCode=0 Jul 09 01:59:37.642979 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:37.642613 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" event={"ID":"4407acfac9b3a400518ad059d4a0580d","Type":"ContainerDied","Data":"d1a54e447fc635a4e06b8458d86b07373fd4465490d977dafdc3f217d3e43f89"} Jul 09 01:59:37.664432 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:37.664374 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-83.ec2.internal" podStartSLOduration=5.664357628 podStartE2EDuration="5.664357628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:35.671198704 +0000 UTC m=+5.559361249" watchObservedRunningTime="2026-07-09 01:59:37.664357628 +0000 UTC m=+7.552520196" Jul 09 01:59:38.575816 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:38.575183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:38.575816 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:38.575201 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:38.575816 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:38.575422 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:38.575816 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:38.575500 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:39.574242 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:39.574199 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:39.574756 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:39.574380 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:40.265688 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:40.265578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:40.265860 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:40.265691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:40.265860 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.265732 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:40.265860 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.265817 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs podName:c847f9eb-ebc3-40a4-a78c-e14cca47a814 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:48.265793674 +0000 UTC m=+18.153956225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs") pod "network-metrics-daemon-t7q22" (UID: "c847f9eb-ebc3-40a4-a78c-e14cca47a814") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:40.265860 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.265825 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:40.266061 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.265884 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret podName:c2f27e32-08c7-4607-9805-e280bcbcd8fe nodeName:}" failed. No retries permitted until 2026-07-09 01:59:48.265866702 +0000 UTC m=+18.154029264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret") pod "global-pull-secret-syncer-fg9k4" (UID: "c2f27e32-08c7-4607-9805-e280bcbcd8fe") : object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:40.367083 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:40.367042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:40.367323 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.367270 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 01:59:40.367323 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.367300 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 01:59:40.367323 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.367314 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lw9l7 for pod openshift-network-diagnostics/network-check-target-jzpk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:40.367492 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.367380 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7 podName:aaa28b10-cbbd-463c-a8a5-e5137a4070f6 nodeName:}" failed. No retries permitted until 2026-07-09 01:59:48.367359381 +0000 UTC m=+18.255521948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw9l7" (UniqueName: "kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7") pod "network-check-target-jzpk6" (UID: "aaa28b10-cbbd-463c-a8a5-e5137a4070f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:40.575583 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:40.575499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:40.576065 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.575624 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:40.576065 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:40.575817 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:40.576065 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:40.575937 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:41.574969 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:41.574459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:41.574969 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:41.574594 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:42.574037 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:42.574005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:42.574546 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:42.574125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:42.574546 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:42.574130 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:42.574546 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:42.574199 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:43.574889 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:43.574846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:43.575395 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:43.574998 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:44.574580 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:44.574501 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:44.574580 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:44.574554 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:44.574926 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:44.574656 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:44.574926 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:44.574791 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:45.574419 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:45.574371 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:45.574572 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:45.574530 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:46.574756 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:46.574722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:46.575193 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:46.574781 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:46.575193 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:46.574873 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:46.575193 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:46.574974 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:47.574774 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:47.574740 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:47.575276 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:47.574871 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:48.331950 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:48.331910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:48.331950 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:48.331955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:48.332182 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.332051 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:48.332182 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.332054 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:48.332182 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.332102 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs podName:c847f9eb-ebc3-40a4-a78c-e14cca47a814 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:04.332089172 +0000 UTC m=+34.220251715 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs") pod "network-metrics-daemon-t7q22" (UID: "c847f9eb-ebc3-40a4-a78c-e14cca47a814") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 01:59:48.332182 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.332114 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret podName:c2f27e32-08c7-4607-9805-e280bcbcd8fe nodeName:}" failed. No retries permitted until 2026-07-09 02:00:04.332108554 +0000 UTC m=+34.220271097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret") pod "global-pull-secret-syncer-fg9k4" (UID: "c2f27e32-08c7-4607-9805-e280bcbcd8fe") : object "kube-system"/"original-pull-secret" not registered Jul 09 01:59:48.432646 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:48.432597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:48.432846 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.432739 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 01:59:48.432846 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.432760 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 01:59:48.432846 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.432775 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lw9l7 for pod openshift-network-diagnostics/network-check-target-jzpk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:48.432846 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.432840 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7 podName:aaa28b10-cbbd-463c-a8a5-e5137a4070f6 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:04.432824515 +0000 UTC m=+34.320987078 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw9l7" (UniqueName: "kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7") pod "network-check-target-jzpk6" (UID: "aaa28b10-cbbd-463c-a8a5-e5137a4070f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 01:59:48.574435 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:48.574350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:48.574591 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:48.574357 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:48.574591 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.574493 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:48.574591 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:48.574531 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:49.574711 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:49.574674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:49.575206 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:49.574789 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:50.575157 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:50.575121 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:50.575157 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:50.575149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:50.575745 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:50.575266 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:50.575745 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:50.575341 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:51.574309 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:51.574278 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:51.574482 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:51.574396 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:52.574856 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:52.574837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:52.575142 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:52.574938 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:52.575142 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:52.575021 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:52.575234 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:52.575140 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:53.574712 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.574286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:53.574881 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:53.574835 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:53.712403 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.712363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" event={"ID":"7dfab8ba-2d59-47e7-abbe-b6695fb06715","Type":"ContainerStarted","Data":"3c04e2fdf140bf82ce8e64e189e463cfabb377c92b9150660caffff4dd895e11"} Jul 09 01:59:53.714330 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.714296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g2rkc" event={"ID":"a6dd0c8d-d57a-428f-9803-3c570479d35e","Type":"ContainerStarted","Data":"44b16550cd702e4945e1e22948e9023b798b911c9ef053aba65aaa726cea31df"} Jul 09 01:59:53.715826 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.715796 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkpgn" event={"ID":"2ab7d58c-8222-4bb3-84b4-a09113c7fb8f","Type":"ContainerStarted","Data":"b57f99fc170a39c9f7e0dbe8b44b153128e85185cf9a67e87f4ade62b6a7603d"} Jul 09 01:59:53.717764 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.717737 2578 generic.go:356] "Generic (PLEG): container finished" podID="3c624a3e-0848-4b58-bb45-a1d1dbb59b8a" containerID="20096c119ef49c1ad8cd9d32dca52409118703370c94e406a1de0cd76ec4476d" exitCode=0 Jul 09 01:59:53.717882 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.717820 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerDied","Data":"20096c119ef49c1ad8cd9d32dca52409118703370c94e406a1de0cd76ec4476d"} Jul 09 01:59:53.721373 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.721343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"b0cc76ab59ec7e7633d3ee79d18974cfdfd4d405f1878d01f71c960bf13df1e9"} Jul 09 01:59:53.721468 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.721455 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"05e84cb244c37455d5c82f518a137eb1e2bcadfe2d7b7603b55c488a5ffc6538"} Jul 09 01:59:53.721539 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.721508 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"60647e58e5229e76223679cd7e741672f55b2cb489b0e79517bbf63acaccea54"} Jul 09 01:59:53.721579 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.721556 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"0df1c37e010d21ae6764fc5656b950f04b10ad7a5abb506043e2fe82de0e86e9"} Jul 09 01:59:53.721622 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.721603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"2e1e5f31c751dc3616ca487021506428a4aad93a66ffa83bb04e0cd3332761ad"} Jul 09 01:59:53.721665 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.721651 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"24fe7d50ddd5443f9a4c68187fd19ef1ae1b57b4d485c3565ee7d510d3742ab1"} Jul 09 01:59:53.722905 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.722878 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w4w95" event={"ID":"70db81f0-b646-4ae3-ab59-02cb9bc52828","Type":"ContainerStarted","Data":"793f114575c539968701b66c91f8e24f69230ccc2952f57bd4809ae8e740ade4"} Jul 09 01:59:53.724434 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.724399 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hg8mp" event={"ID":"e1f20951-58ef-4818-8715-c74ff5029e4d","Type":"ContainerStarted","Data":"6c6e2bb71bcce2dfc356ef0bdad6b14f4363e99ccd7c0f3b420e935f57dec794"} Jul 09 01:59:53.726172 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.726147 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" event={"ID":"4407acfac9b3a400518ad059d4a0580d","Type":"ContainerStarted","Data":"a40d02b984d9fe86c3afe50480246fad92c41566c1c9b45426aaa7d908d20b4d"} Jul 09 01:59:53.727650 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.727629 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" event={"ID":"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd","Type":"ContainerStarted","Data":"027fd96d0da119d63936b3761bdaac57736e8170f74738502cf669063ecbc968"} Jul 09 01:59:53.779458 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.779391 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g2rkc" podStartSLOduration=2.986881076 podStartE2EDuration="22.779368704s" totalImagesPullingTime="19.792487628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:53.778624061 +0000 UTC m=+23.666786627" watchObservedRunningTime="2026-07-09 01:59:53.779368704 +0000 UTC m=+23.667531270" Jul 09 01:59:53.780218 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.780175 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mq8t6" podStartSLOduration=3.052345951 podStartE2EDuration="22.780162682s" totalImagesPullingTime="19.727816731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:53.740592005 +0000 UTC m=+23.628754570" watchObservedRunningTime="2026-07-09 01:59:53.780162682 +0000 UTC m=+23.668325249" Jul 09 01:59:53.835391 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.835331 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hg8mp" podStartSLOduration=3.055878197 podStartE2EDuration="22.83530882s" totalImagesPullingTime="19.779430623s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:53.834699735 +0000 UTC m=+23.722862300" watchObservedRunningTime="2026-07-09 01:59:53.83530882 +0000 UTC m=+23.723471386" Jul 09 01:59:53.849811 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.849749 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w4w95" podStartSLOduration=3.078264078 podStartE2EDuration="22.849728138s" totalImagesPullingTime="19.77146406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:53.849483473 +0000 UTC m=+23.737646040" watchObservedRunningTime="2026-07-09 01:59:53.849728138 +0000 UTC m=+23.737890706" Jul 09 01:59:53.866105 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.866048 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vkpgn" podStartSLOduration=3.12076725 podStartE2EDuration="22.866026886s" totalImagesPullingTime="19.745259636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:53.86588056 +0000 UTC m=+23.754043143" watchObservedRunningTime="2026-07-09 01:59:53.866026886 +0000 UTC m=+23.754189455" Jul 09 01:59:53.886904 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:53.886844 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-83.ec2.internal" podStartSLOduration=21.886826781 podStartE2EDuration="21.886826781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:53.886570564 +0000 UTC m=+23.774733130" watchObservedRunningTime="2026-07-09 01:59:53.886826781 +0000 UTC m=+23.774989346" Jul 09 01:59:54.212603 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.212573 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 01:59:54.551898 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.551786 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T01:59:54.21258994Z","UUID":"8f566fde-1035-4a68-a0ca-474af788e0c2","Handler":null,"Name":"","Endpoint":""} Jul 09 01:59:54.553401 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.553367 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 01:59:54.553539 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.553417 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 01:59:54.574681 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.574646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:54.574877 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.574646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:54.574877 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:54.574781 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:54.574877 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:54.574855 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:54.732007 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.731972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" event={"ID":"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd","Type":"ContainerStarted","Data":"21e338899bdc7ec21921c9d301ae2396a2ac92b3ca8ef0f4c531558087b680e6"} Jul 09 01:59:54.733473 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.733438 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdk2r" event={"ID":"1b17fef7-10c2-4ebd-9f80-0f3c7417b0d3","Type":"ContainerStarted","Data":"099a68f90f2d8d39e4f7d1f28e73b4212e0cdaf71abe95115a1d2f3189c9cb50"} Jul 09 01:59:54.753393 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:54.753343 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zdk2r" podStartSLOduration=4.041184473 podStartE2EDuration="23.753328695s" totalImagesPullingTime="19.712144222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:54.753126969 +0000 UTC m=+24.641289533" watchObservedRunningTime="2026-07-09 01:59:54.753328695 +0000 UTC m=+24.641491259" Jul 09 01:59:55.575119 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:55.575083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:55.575487 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:55.575260 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:55.739364 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:55.739169 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"79e878de7a88db00d135900e3b2a4726de11e57837df01bf419ffc80dd5a1dd4"} Jul 09 01:59:56.575127 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:56.575089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:56.575586 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:56.575089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:56.575586 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:56.575209 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:56.575586 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:56.575339 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:56.745460 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:56.745421 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" event={"ID":"59ecb41d-4d2b-4e4b-9d31-2d4b668dbfdd","Type":"ContainerStarted","Data":"086b0c2e383fe093c456460a9176118fb6670de69232a9fe237a5a2f50db612d"} Jul 09 01:59:57.574373 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:57.574336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:57.574528 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:57.574464 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 01:59:57.751810 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:57.751664 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" event={"ID":"1099b323-984e-47ae-9569-24c13d735e17","Type":"ContainerStarted","Data":"e244e371f9b92e8395c1bffd69ad3228c545851d58552f50b0849e34bc791c55"} Jul 09 01:59:57.817020 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:57.816957 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tqglh" podStartSLOduration=4.400269465 podStartE2EDuration="26.816932674s" totalImagesPullingTime="22.416663209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:56.769566333 +0000 UTC m=+26.657728898" watchObservedRunningTime="2026-07-09 01:59:57.816932674 +0000 UTC m=+27.705095243" Jul 09 01:59:57.817487 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:57.817445 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" podStartSLOduration=6.995820967 podStartE2EDuration="26.817430773s" totalImagesPullingTime="19.821609806s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 01:59:57.816783558 +0000 UTC m=+27.704946139" watchObservedRunningTime="2026-07-09 01:59:57.817430773 +0000 UTC m=+27.705593342" Jul 09 01:59:58.563621 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.563377 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:58.564138 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.564094 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:58.574221 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.574190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 01:59:58.574376 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.574202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 01:59:58.574376 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:58.574331 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 01:59:58.574474 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:58.574438 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 01:59:58.755311 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.755278 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:58.755797 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.755367 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:58.755797 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.755395 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:58.755797 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.755481 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:58.755797 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.755560 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vkpgn" Jul 09 01:59:58.772899 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.772867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:58.773083 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:58.773061 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 01:59:59.574695 ip-10-0-131-83 kubenswrapper[2578]: I0709 01:59:59.574659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 01:59:59.574875 ip-10-0-131-83 kubenswrapper[2578]: E0709 01:59:59.574794 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 02:00:00.042292 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:00.042232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fg9k4"] Jul 09 02:00:00.042723 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:00.042368 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 02:00:00.042723 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:00.042457 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 02:00:00.046372 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:00.046341 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t7q22"] Jul 09 02:00:00.046527 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:00.046478 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 02:00:00.046593 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:00.046576 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 02:00:00.051288 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:00.051263 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jzpk6"] Jul 09 02:00:00.051437 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:00.051349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:00.051484 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:00.051442 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 02:00:00.761296 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:00.761240 2578 generic.go:356] "Generic (PLEG): container finished" podID="3c624a3e-0848-4b58-bb45-a1d1dbb59b8a" containerID="fa824d51cdbaaa80d79fa1e37900c8fe4026851ec22abf87cf7829d07e3303a9" exitCode=0 Jul 09 02:00:00.761296 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:00.761287 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerDied","Data":"fa824d51cdbaaa80d79fa1e37900c8fe4026851ec22abf87cf7829d07e3303a9"} Jul 09 02:00:01.575074 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:01.575035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 02:00:01.575486 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:01.575092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 02:00:01.575486 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:01.575104 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:01.575486 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:01.575165 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 02:00:01.575486 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:01.575268 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 02:00:01.575486 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:01.575361 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 02:00:02.768002 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:02.767755 2578 generic.go:356] "Generic (PLEG): container finished" podID="3c624a3e-0848-4b58-bb45-a1d1dbb59b8a" containerID="5ea1aff87a91a8bac82e3a1f6005480798775e514ad13de9be03c54a1b8e7fd0" exitCode=0 Jul 09 02:00:02.768002 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:02.767839 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerDied","Data":"5ea1aff87a91a8bac82e3a1f6005480798775e514ad13de9be03c54a1b8e7fd0"} Jul 09 02:00:03.574366 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:03.574333 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 02:00:03.574532 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:03.574334 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:03.574532 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:03.574454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fg9k4" podUID="c2f27e32-08c7-4607-9805-e280bcbcd8fe" Jul 09 02:00:03.574532 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:03.574512 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jzpk6" podUID="aaa28b10-cbbd-463c-a8a5-e5137a4070f6" Jul 09 02:00:03.574698 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:03.574330 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 02:00:03.574698 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:03.574626 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t7q22" podUID="c847f9eb-ebc3-40a4-a78c-e14cca47a814" Jul 09 02:00:04.346729 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:04.346689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 02:00:04.347281 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:04.346761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 02:00:04.347281 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:04.346835 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 02:00:04.347281 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:04.346897 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs podName:c847f9eb-ebc3-40a4-a78c-e14cca47a814 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:36.346882099 +0000 UTC m=+66.235044642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs") pod "network-metrics-daemon-t7q22" (UID: "c847f9eb-ebc3-40a4-a78c-e14cca47a814") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 02:00:04.347281 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:04.346841 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 02:00:04.347281 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:04.346967 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret podName:c2f27e32-08c7-4607-9805-e280bcbcd8fe nodeName:}" failed. No retries permitted until 2026-07-09 02:00:36.346954268 +0000 UTC m=+66.235116832 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret") pod "global-pull-secret-syncer-fg9k4" (UID: "c2f27e32-08c7-4607-9805-e280bcbcd8fe") : object "kube-system"/"original-pull-secret" not registered Jul 09 02:00:04.447789 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:04.447756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:04.447964 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:04.447898 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 02:00:04.447964 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:04.447920 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 02:00:04.447964 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:04.447930 2578 projected.go:196] Error preparing data for projected volume kube-api-access-lw9l7 for pod openshift-network-diagnostics/network-check-target-jzpk6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 02:00:04.448062 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:04.447980 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7 podName:aaa28b10-cbbd-463c-a8a5-e5137a4070f6 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:36.447965057 +0000 UTC m=+66.336127600 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lw9l7" (UniqueName: "kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7") pod "network-check-target-jzpk6" (UID: "aaa28b10-cbbd-463c-a8a5-e5137a4070f6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 02:00:04.773883 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:04.773854 2578 generic.go:356] "Generic (PLEG): container finished" podID="3c624a3e-0848-4b58-bb45-a1d1dbb59b8a" containerID="5d2f84344563b9387b3a3f8cfd4ee67195ab7dc5b6e811cfc289cd5172502047" exitCode=0 Jul 09 02:00:04.774060 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:04.773902 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerDied","Data":"5d2f84344563b9387b3a3f8cfd4ee67195ab7dc5b6e811cfc289cd5172502047"} Jul 09 02:00:05.453086 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.453011 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-83.ec2.internal" event="NodeReady" Jul 09 02:00:05.453451 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.453163 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 02:00:05.490188 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.490157 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64b9d58d4f-zkvx7"] Jul 09 02:00:05.495391 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.495363 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-n8hg2"] Jul 09 02:00:05.495543 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.495427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.497655 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.497636 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt"] Jul 09 02:00:05.497776 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.497759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.499117 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.499101 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 02:00:05.499426 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.499409 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hmfxw\"" Jul 09 02:00:05.499543 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.499487 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 02:00:05.499645 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.499626 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 02:00:05.499890 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.499872 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-789b84dcd9-wrbcr"] Jul 09 02:00:05.500031 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.499941 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.500233 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.500216 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 02:00:05.500508 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.500493 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 09 02:00:05.500571 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.500504 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 02:00:05.500620 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.500505 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9znql\"" Jul 09 02:00:05.500680 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.500666 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 09 02:00:05.502292 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.502277 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.504371 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.504341 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 09 02:00:05.504371 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.504366 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-flcfn\"" Jul 09 02:00:05.504526 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.504371 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 02:00:05.504608 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.504591 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 02:00:05.506288 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.506272 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 09 02:00:05.507343 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.507323 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 02:00:05.508800 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.508779 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 02:00:05.508901 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.508813 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 02:00:05.508901 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.508848 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 02:00:05.509206 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.509181 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 02:00:05.509328 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.509301 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nnpdv\"" Jul 09 02:00:05.510158 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.510139 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 02:00:05.514849 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.514827 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 09 02:00:05.515261 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.515228 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64b9d58d4f-zkvx7"] Jul 09 02:00:05.515539 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.515517 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 02:00:05.515998 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.515980 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-n8hg2"] Jul 09 02:00:05.516799 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.516782 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt"] Jul 09 02:00:05.517649 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.517615 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-789b84dcd9-wrbcr"] Jul 09 02:00:05.518448 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.518423 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-56q8v"] Jul 09 02:00:05.522215 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.522192 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2qgxg"] Jul 09 02:00:05.522327 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.522314 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.524759 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.524742 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 02:00:05.524920 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.524901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2qgxg" Jul 09 02:00:05.525488 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.525467 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 02:00:05.525577 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.525489 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nxtt\"" Jul 09 02:00:05.527918 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.527896 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 02:00:05.528221 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.528140 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dnxq9\"" Jul 09 02:00:05.528735 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.528714 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 02:00:05.529022 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.528207 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 02:00:05.530690 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.530637 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-56q8v"] Jul 09 02:00:05.532662 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.532641 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2qgxg"] Jul 09 02:00:05.574078 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.574042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 02:00:05.574276 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.574042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 02:00:05.574276 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.574042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:05.576796 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.576770 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 02:00:05.576981 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.576880 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 02:00:05.577171 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.577153 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 02:00:05.577272 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.577176 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 02:00:05.577272 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.577153 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j65lj\"" Jul 09 02:00:05.577272 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.577228 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmqn4\"" Jul 09 02:00:05.655232 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-tmp-dir\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.655232 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655240 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-certificates\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.655531 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-config-volume\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.655531 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abb47d9b-697a-4099-8423-03edd6d52ad6-cert\") pod \"ingress-canary-2qgxg\" (UID: \"abb47d9b-697a-4099-8423-03edd6d52ad6\") " pod="openshift-ingress-canary/ingress-canary-2qgxg" Jul 09 02:00:05.655531 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhl9\" (UniqueName: \"kubernetes.io/projected/abb47d9b-697a-4099-8423-03edd6d52ad6-kube-api-access-5hhl9\") pod \"ingress-canary-2qgxg\" (UID: \"abb47d9b-697a-4099-8423-03edd6d52ad6\") " pod="openshift-ingress-canary/ingress-canary-2qgxg" Jul 09 02:00:05.655531 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.655531 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d66a9ae-2d58-4260-a16d-096a4311a43a-ca-trust-extracted\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.655531 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.655531 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.655797 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13021762-a6ad-46ab-a4de-dc1838a6662e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.655797 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13021762-a6ad-46ab-a4de-dc1838a6662e-serving-cert\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.655797 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13021762-a6ad-46ab-a4de-dc1838a6662e-tmp\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.655797 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dp5j\" (UniqueName: \"kubernetes.io/projected/13021762-a6ad-46ab-a4de-dc1838a6662e-kube-api-access-8dp5j\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.655797 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczwl\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-kube-api-access-pczwl\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.655797 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-image-registry-private-configuration\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.655797 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655761 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-stats-auth\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.655797 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-bound-sa-token\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-default-certificate\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk6kz\" (UniqueName: \"kubernetes.io/projected/a98ef70b-a070-45fd-910e-0353efdd0278-kube-api-access-tk6kz\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzbl\" (UniqueName: \"kubernetes.io/projected/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-kube-api-access-gtzbl\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655906 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdf6\" (UniqueName: \"kubernetes.io/projected/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-kube-api-access-6hdf6\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/13021762-a6ad-46ab-a4de-dc1838a6662e-snapshots\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.655964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.656009 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-installation-pull-secrets\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.656075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.656040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13021762-a6ad-46ab-a4de-dc1838a6662e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.656389 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.656084 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-trusted-ca\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.656389 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.656123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.756541 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756506 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13021762-a6ad-46ab-a4de-dc1838a6662e-serving-cert\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.756541 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13021762-a6ad-46ab-a4de-dc1838a6662e-tmp\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dp5j\" (UniqueName: \"kubernetes.io/projected/13021762-a6ad-46ab-a4de-dc1838a6662e-kube-api-access-8dp5j\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczwl\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-kube-api-access-pczwl\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756630 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-image-registry-private-configuration\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-stats-auth\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-bound-sa-token\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-default-certificate\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk6kz\" (UniqueName: \"kubernetes.io/projected/a98ef70b-a070-45fd-910e-0353efdd0278-kube-api-access-tk6kz\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.756782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzbl\" (UniqueName: \"kubernetes.io/projected/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-kube-api-access-gtzbl\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdf6\" (UniqueName: \"kubernetes.io/projected/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-kube-api-access-6hdf6\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/13021762-a6ad-46ab-a4de-dc1838a6662e-snapshots\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.756840 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.756861 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b9d58d4f-zkvx7: secret "image-registry-tls" not found Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-installation-pull-secrets\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.756928 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls podName:6d66a9ae-2d58-4260-a16d-096a4311a43a nodeName:}" failed. No retries permitted until 2026-07-09 02:00:06.25690936 +0000 UTC m=+36.145071907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls") pod "image-registry-64b9d58d4f-zkvx7" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a") : secret "image-registry-tls" not found Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756969 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13021762-a6ad-46ab-a4de-dc1838a6662e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.756999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-trusted-ca\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.757167 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.757004 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13021762-a6ad-46ab-a4de-dc1838a6662e-tmp\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.757983 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-trusted-ca\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-tmp-dir\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-certificates\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-config-volume\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abb47d9b-697a-4099-8423-03edd6d52ad6-cert\") pod \"ingress-canary-2qgxg\" (UID: \"abb47d9b-697a-4099-8423-03edd6d52ad6\") " pod="openshift-ingress-canary/ingress-canary-2qgxg" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhl9\" (UniqueName: \"kubernetes.io/projected/abb47d9b-697a-4099-8423-03edd6d52ad6-kube-api-access-5hhl9\") pod \"ingress-canary-2qgxg\" (UID: \"abb47d9b-697a-4099-8423-03edd6d52ad6\") " pod="openshift-ingress-canary/ingress-canary-2qgxg" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d66a9ae-2d58-4260-a16d-096a4311a43a-ca-trust-extracted\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13021762-a6ad-46ab-a4de-dc1838a6662e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13021762-a6ad-46ab-a4de-dc1838a6662e-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.758606 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/13021762-a6ad-46ab-a4de-dc1838a6662e-snapshots\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.758654 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.758703 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 02:00:05.759361 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.758723 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls podName:50bc272c-2f83-44a1-9c62-6cbbd2e5a473 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:06.258705262 +0000 UTC m=+36.146867815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls") pod "dns-default-56q8v" (UID: "50bc272c-2f83-44a1-9c62-6cbbd2e5a473") : secret "dns-default-metrics-tls" not found Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.758759 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:06.2587446 +0000 UTC m=+36.146907172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : secret "router-metrics-certs-default" not found Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.758839 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:06.258829636 +0000 UTC m=+36.146992220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : configmap references non-existent config key: service-ca.crt Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.759098 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-config-volume\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.759131 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d66a9ae-2d58-4260-a16d-096a4311a43a-ca-trust-extracted\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.759223 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.759276 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:05.759290 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls podName:fb4e10fa-e512-499f-9cb3-9b98ed8bcba7 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:06.259272711 +0000 UTC m=+36.147435278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-72vjt" (UID: "fb4e10fa-e512-499f-9cb3-9b98ed8bcba7") : secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.759219 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13021762-a6ad-46ab-a4de-dc1838a6662e-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.759417 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-tmp-dir\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.760333 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.759732 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-certificates\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.761540 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.761519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13021762-a6ad-46ab-a4de-dc1838a6662e-serving-cert\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.761663 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.761566 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-image-registry-private-configuration\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.761713 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.761657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-installation-pull-secrets\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.761748 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.761710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-stats-auth\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.761781 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.761740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-default-certificate\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.761868 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.761851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abb47d9b-697a-4099-8423-03edd6d52ad6-cert\") pod \"ingress-canary-2qgxg\" (UID: \"abb47d9b-697a-4099-8423-03edd6d52ad6\") " pod="openshift-ingress-canary/ingress-canary-2qgxg" Jul 09 02:00:05.767169 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.767143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczwl\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-kube-api-access-pczwl\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.767377 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.767356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdf6\" (UniqueName: \"kubernetes.io/projected/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-kube-api-access-6hdf6\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:05.767492 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.767471 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-bound-sa-token\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:05.767560 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.767514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk6kz\" (UniqueName: \"kubernetes.io/projected/a98ef70b-a070-45fd-910e-0353efdd0278-kube-api-access-tk6kz\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:05.767817 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.767786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzbl\" (UniqueName: \"kubernetes.io/projected/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-kube-api-access-gtzbl\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:05.768104 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.768086 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhl9\" (UniqueName: \"kubernetes.io/projected/abb47d9b-697a-4099-8423-03edd6d52ad6-kube-api-access-5hhl9\") pod \"ingress-canary-2qgxg\" (UID: \"abb47d9b-697a-4099-8423-03edd6d52ad6\") " pod="openshift-ingress-canary/ingress-canary-2qgxg" Jul 09 02:00:05.768462 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.768443 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dp5j\" (UniqueName: \"kubernetes.io/projected/13021762-a6ad-46ab-a4de-dc1838a6662e-kube-api-access-8dp5j\") pod \"insights-operator-5c847dcfc4-n8hg2\" (UID: \"13021762-a6ad-46ab-a4de-dc1838a6662e\") " pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.814772 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.814728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" Jul 09 02:00:05.843645 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.843609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2qgxg" Jul 09 02:00:05.988365 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.988163 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-n8hg2"] Jul 09 02:00:05.991585 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:05.991562 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2qgxg"] Jul 09 02:00:05.992429 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:05.992407 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13021762_a6ad_46ab_a4de_dc1838a6662e.slice/crio-806f84528b97664f27643dd2d2b24450ab86bd162136be4ecf64290bc4b01857 WatchSource:0}: Error finding container 806f84528b97664f27643dd2d2b24450ab86bd162136be4ecf64290bc4b01857: Status 404 returned error can't find the container with id 806f84528b97664f27643dd2d2b24450ab86bd162136be4ecf64290bc4b01857 Jul 09 02:00:05.993862 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:05.993843 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb47d9b_697a_4099_8423_03edd6d52ad6.slice/crio-bd020b7a4173a4c68ed8b66ec3e89eac85b909647a5b8672b3d9125e18065018 WatchSource:0}: Error finding container bd020b7a4173a4c68ed8b66ec3e89eac85b909647a5b8672b3d9125e18065018: Status 404 returned error can't find the container with id bd020b7a4173a4c68ed8b66ec3e89eac85b909647a5b8672b3d9125e18065018 Jul 09 02:00:06.264169 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:06.264132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:06.264342 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:06.264176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:06.264342 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264304 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 02:00:06.264342 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264323 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b9d58d4f-zkvx7: secret "image-registry-tls" not found Jul 09 02:00:06.264342 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264335 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 02:00:06.264470 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264384 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls podName:6d66a9ae-2d58-4260-a16d-096a4311a43a nodeName:}" failed. No retries permitted until 2026-07-09 02:00:07.264367127 +0000 UTC m=+37.152529670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls") pod "image-registry-64b9d58d4f-zkvx7" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a") : secret "image-registry-tls" not found Jul 09 02:00:06.264470 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264400 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:07.264391885 +0000 UTC m=+37.152554427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : secret "router-metrics-certs-default" not found Jul 09 02:00:06.264470 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:06.264451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:06.264588 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:06.264474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:06.264588 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:06.264492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:06.264588 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264577 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 02:00:06.264588 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264584 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:06.264727 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264619 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls podName:50bc272c-2f83-44a1-9c62-6cbbd2e5a473 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:07.264605908 +0000 UTC m=+37.152768452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls") pod "dns-default-56q8v" (UID: "50bc272c-2f83-44a1-9c62-6cbbd2e5a473") : secret "dns-default-metrics-tls" not found Jul 09 02:00:06.264727 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264632 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls podName:fb4e10fa-e512-499f-9cb3-9b98ed8bcba7 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:07.264626103 +0000 UTC m=+37.152788645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-72vjt" (UID: "fb4e10fa-e512-499f-9cb3-9b98ed8bcba7") : secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:06.264727 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:06.264644 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:07.264638495 +0000 UTC m=+37.152801038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : configmap references non-existent config key: service-ca.crt Jul 09 02:00:06.789856 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:06.781619 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2qgxg" event={"ID":"abb47d9b-697a-4099-8423-03edd6d52ad6","Type":"ContainerStarted","Data":"bd020b7a4173a4c68ed8b66ec3e89eac85b909647a5b8672b3d9125e18065018"} Jul 09 02:00:06.789856 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:06.783095 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" event={"ID":"13021762-a6ad-46ab-a4de-dc1838a6662e","Type":"ContainerStarted","Data":"806f84528b97664f27643dd2d2b24450ab86bd162136be4ecf64290bc4b01857"} Jul 09 02:00:07.274535 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.274490 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:07.274750 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.274551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:07.274750 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.274583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:07.274750 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.274659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:07.274750 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.274694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:07.274750 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.274745 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 02:00:07.274989 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.274830 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 02:00:07.274989 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.274854 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls podName:50bc272c-2f83-44a1-9c62-6cbbd2e5a473 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:09.274834825 +0000 UTC m=+39.162997378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls") pod "dns-default-56q8v" (UID: "50bc272c-2f83-44a1-9c62-6cbbd2e5a473") : secret "dns-default-metrics-tls" not found Jul 09 02:00:07.274989 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.274874 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:09.274864598 +0000 UTC m=+39.163027141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : secret "router-metrics-certs-default" not found Jul 09 02:00:07.274989 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.274908 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:09.274897777 +0000 UTC m=+39.163060319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : configmap references non-existent config key: service-ca.crt Jul 09 02:00:07.274989 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.274957 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:07.274989 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.274986 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls podName:fb4e10fa-e512-499f-9cb3-9b98ed8bcba7 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:09.274976618 +0000 UTC m=+39.163139161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-72vjt" (UID: "fb4e10fa-e512-499f-9cb3-9b98ed8bcba7") : secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:07.275289 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.275038 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 02:00:07.275289 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.275048 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b9d58d4f-zkvx7: secret "image-registry-tls" not found Jul 09 02:00:07.275289 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.275074 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls podName:6d66a9ae-2d58-4260-a16d-096a4311a43a nodeName:}" failed. No retries permitted until 2026-07-09 02:00:09.275065072 +0000 UTC m=+39.163227618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls") pod "image-registry-64b9d58d4f-zkvx7" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a") : secret "image-registry-tls" not found Jul 09 02:00:07.603523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.603429 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7"] Jul 09 02:00:07.621635 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.621581 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7"] Jul 09 02:00:07.621815 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.621720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:07.626516 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.626165 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 02:00:07.626516 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.626275 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nn8t6\"" Jul 09 02:00:07.626516 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.626180 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 02:00:07.628063 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.627425 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 02:00:07.708988 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.708948 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p"] Jul 09 02:00:07.725849 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.725804 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv"] Jul 09 02:00:07.726072 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.725930 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.729295 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.729228 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 02:00:07.729522 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.729502 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 02:00:07.729717 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.729698 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 02:00:07.730121 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.729926 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5b6t7\"" Jul 09 02:00:07.730262 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.730224 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 02:00:07.745883 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.745838 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-4mkwb"] Jul 09 02:00:07.746025 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.745974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv" Jul 09 02:00:07.749284 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.749025 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vt8f6\"" Jul 09 02:00:07.749284 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.749022 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 09 02:00:07.749284 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.749094 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 09 02:00:07.768790 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.768763 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv"] Jul 09 02:00:07.768790 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.768798 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p"] Jul 09 02:00:07.769016 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.768902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.769016 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.768933 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-4mkwb"] Jul 09 02:00:07.772130 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.772110 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 09 02:00:07.772130 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.772116 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 09 02:00:07.772344 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.772112 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 09 02:00:07.772471 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.772455 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 09 02:00:07.772719 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.772687 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-fmzkg\"" Jul 09 02:00:07.777625 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.777599 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 09 02:00:07.780148 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.780126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brjxm\" (UniqueName: \"kubernetes.io/projected/a3cca3f2-372d-47d6-9d00-a21c7e187716-kube-api-access-brjxm\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:07.780274 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.780161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:07.804419 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.804387 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v"] Jul 09 02:00:07.824990 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.824957 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v"] Jul 09 02:00:07.825162 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.825097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:07.828145 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.828118 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 02:00:07.828319 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.828125 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 02:00:07.828439 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.828418 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 02:00:07.828559 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.828456 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-879j4\"" Jul 09 02:00:07.828629 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.828572 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 02:00:07.881345 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881236 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d84a9c83-d058-4182-814d-5e9a282d2013-config\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.881345 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzmd\" (UniqueName: \"kubernetes.io/projected/98d79bfc-4203-478c-bf3f-c6bac300e131-kube-api-access-kgzmd\") pod \"volume-data-source-validator-5d456fdbcd-dg9nv\" (UID: \"98d79bfc-4203-478c-bf3f-c6bac300e131\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv" Jul 09 02:00:07.881345 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:07.881619 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881423 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d84a9c83-d058-4182-814d-5e9a282d2013-trusted-ca\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.881619 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprcl\" (UniqueName: \"kubernetes.io/projected/71731d5b-3313-4046-8579-f828a13ba12c-kube-api-access-qprcl\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.881619 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.881482 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 02:00:07.881619 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:07.881544 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls podName:a3cca3f2-372d-47d6-9d00-a21c7e187716 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:08.381525246 +0000 UTC m=+38.269687791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-94cb7" (UID: "a3cca3f2-372d-47d6-9d00-a21c7e187716") : secret "samples-operator-tls" not found Jul 09 02:00:07.881619 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71731d5b-3313-4046-8579-f828a13ba12c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.881908 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94j78\" (UniqueName: \"kubernetes.io/projected/d84a9c83-d058-4182-814d-5e9a282d2013-kube-api-access-94j78\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.881908 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881655 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d84a9c83-d058-4182-814d-5e9a282d2013-serving-cert\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.881908 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71731d5b-3313-4046-8579-f828a13ba12c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.881908 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.881827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brjxm\" (UniqueName: \"kubernetes.io/projected/a3cca3f2-372d-47d6-9d00-a21c7e187716-kube-api-access-brjxm\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:07.897335 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.897308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brjxm\" (UniqueName: \"kubernetes.io/projected/a3cca3f2-372d-47d6-9d00-a21c7e187716-kube-api-access-brjxm\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:07.983199 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d84a9c83-d058-4182-814d-5e9a282d2013-config\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.983199 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983202 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzmd\" (UniqueName: \"kubernetes.io/projected/98d79bfc-4203-478c-bf3f-c6bac300e131-kube-api-access-kgzmd\") pod \"volume-data-source-validator-5d456fdbcd-dg9nv\" (UID: \"98d79bfc-4203-478c-bf3f-c6bac300e131\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv" Jul 09 02:00:07.983466 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d84a9c83-d058-4182-814d-5e9a282d2013-trusted-ca\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.983466 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprcl\" (UniqueName: \"kubernetes.io/projected/71731d5b-3313-4046-8579-f828a13ba12c-kube-api-access-qprcl\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.983466 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8290af00-40ea-4657-bf4e-92137e8bc739-config\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:07.983466 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71731d5b-3313-4046-8579-f828a13ba12c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.983466 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983435 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94j78\" (UniqueName: \"kubernetes.io/projected/d84a9c83-d058-4182-814d-5e9a282d2013-kube-api-access-94j78\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.983719 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983476 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d84a9c83-d058-4182-814d-5e9a282d2013-serving-cert\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.983719 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8290af00-40ea-4657-bf4e-92137e8bc739-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:07.983719 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71731d5b-3313-4046-8579-f828a13ba12c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.983719 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.983633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbk9f\" (UniqueName: \"kubernetes.io/projected/8290af00-40ea-4657-bf4e-92137e8bc739-kube-api-access-nbk9f\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:07.984203 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.984031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/71731d5b-3313-4046-8579-f828a13ba12c-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.984354 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.984331 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d84a9c83-d058-4182-814d-5e9a282d2013-config\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.984439 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.984415 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d84a9c83-d058-4182-814d-5e9a282d2013-trusted-ca\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.986810 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.986779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/71731d5b-3313-4046-8579-f828a13ba12c-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:07.986888 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.986829 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d84a9c83-d058-4182-814d-5e9a282d2013-serving-cert\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.993668 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.993633 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94j78\" (UniqueName: \"kubernetes.io/projected/d84a9c83-d058-4182-814d-5e9a282d2013-kube-api-access-94j78\") pod \"console-operator-68b757fb67-4mkwb\" (UID: \"d84a9c83-d058-4182-814d-5e9a282d2013\") " pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:07.993819 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.993631 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzmd\" (UniqueName: \"kubernetes.io/projected/98d79bfc-4203-478c-bf3f-c6bac300e131-kube-api-access-kgzmd\") pod \"volume-data-source-validator-5d456fdbcd-dg9nv\" (UID: \"98d79bfc-4203-478c-bf3f-c6bac300e131\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv" Jul 09 02:00:07.994197 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:07.994052 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprcl\" (UniqueName: \"kubernetes.io/projected/71731d5b-3313-4046-8579-f828a13ba12c-kube-api-access-qprcl\") pod \"kube-storage-version-migrator-operator-846bf668f7-htm8p\" (UID: \"71731d5b-3313-4046-8579-f828a13ba12c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:08.040876 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.040834 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" Jul 09 02:00:08.057592 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.057561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv" Jul 09 02:00:08.080439 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.080404 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:08.084876 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.084846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8290af00-40ea-4657-bf4e-92137e8bc739-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:08.085043 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.084890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbk9f\" (UniqueName: \"kubernetes.io/projected/8290af00-40ea-4657-bf4e-92137e8bc739-kube-api-access-nbk9f\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:08.085043 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.085007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8290af00-40ea-4657-bf4e-92137e8bc739-config\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:08.085905 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.085863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8290af00-40ea-4657-bf4e-92137e8bc739-config\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:08.088058 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.088033 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8290af00-40ea-4657-bf4e-92137e8bc739-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:08.095478 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.095417 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbk9f\" (UniqueName: \"kubernetes.io/projected/8290af00-40ea-4657-bf4e-92137e8bc739-kube-api-access-nbk9f\") pod \"service-ca-operator-5d46d9ffd4-4x69v\" (UID: \"8290af00-40ea-4657-bf4e-92137e8bc739\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:08.137355 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.137274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" Jul 09 02:00:08.388389 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:08.388305 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:08.388583 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:08.388479 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 02:00:08.388583 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:08.388565 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls podName:a3cca3f2-372d-47d6-9d00-a21c7e187716 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:09.388542015 +0000 UTC m=+39.276704582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-94cb7" (UID: "a3cca3f2-372d-47d6-9d00-a21c7e187716") : secret "samples-operator-tls" not found Jul 09 02:00:09.298488 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:09.298443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:09.298507 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:09.298534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:09.298632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298668 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:13.298648624 +0000 UTC m=+43.186811183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : configmap references non-existent config key: service-ca.crt Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:09.298710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298727 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298729 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298794 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls podName:fb4e10fa-e512-499f-9cb3-9b98ed8bcba7 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:13.298780675 +0000 UTC m=+43.186943217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-72vjt" (UID: "fb4e10fa-e512-499f-9cb3-9b98ed8bcba7") : secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298739 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b9d58d4f-zkvx7: secret "image-registry-tls" not found Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298833 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls podName:6d66a9ae-2d58-4260-a16d-096a4311a43a nodeName:}" failed. No retries permitted until 2026-07-09 02:00:13.298822533 +0000 UTC m=+43.186985095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls") pod "image-registry-64b9d58d4f-zkvx7" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a") : secret "image-registry-tls" not found Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298850 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298884 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298889 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:13.298878355 +0000 UTC m=+43.187040904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : secret "router-metrics-certs-default" not found Jul 09 02:00:09.299008 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.298950 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls podName:50bc272c-2f83-44a1-9c62-6cbbd2e5a473 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:13.298937843 +0000 UTC m=+43.187100386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls") pod "dns-default-56q8v" (UID: "50bc272c-2f83-44a1-9c62-6cbbd2e5a473") : secret "dns-default-metrics-tls" not found Jul 09 02:00:09.399512 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:09.399467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:09.399678 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.399592 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 02:00:09.399678 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:09.399648 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls podName:a3cca3f2-372d-47d6-9d00-a21c7e187716 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:11.399635565 +0000 UTC m=+41.287798108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-94cb7" (UID: "a3cca3f2-372d-47d6-9d00-a21c7e187716") : secret "samples-operator-tls" not found Jul 09 02:00:11.416003 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.415960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:11.416393 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:11.416118 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 02:00:11.416393 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:11.416182 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls podName:a3cca3f2-372d-47d6-9d00-a21c7e187716 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:15.4161685 +0000 UTC m=+45.304331060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-94cb7" (UID: "a3cca3f2-372d-47d6-9d00-a21c7e187716") : secret "samples-operator-tls" not found Jul 09 02:00:11.685826 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.685796 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p"] Jul 09 02:00:11.697642 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:11.697612 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71731d5b_3313_4046_8579_f828a13ba12c.slice/crio-4dbbff99107e8c4bd1ae14f3045da0e73dee02960c29cb3ea855cf504e97215e WatchSource:0}: Error finding container 4dbbff99107e8c4bd1ae14f3045da0e73dee02960c29cb3ea855cf504e97215e: Status 404 returned error can't find the container with id 4dbbff99107e8c4bd1ae14f3045da0e73dee02960c29cb3ea855cf504e97215e Jul 09 02:00:11.736425 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.736393 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv"] Jul 09 02:00:11.740360 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:11.740329 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d79bfc_4203_478c_bf3f_c6bac300e131.slice/crio-ac82fbdea9f0b0449657ad360ce5483e6924ecebeb4393ca4329442f61f5f9e5 WatchSource:0}: Error finding container ac82fbdea9f0b0449657ad360ce5483e6924ecebeb4393ca4329442f61f5f9e5: Status 404 returned error can't find the container with id ac82fbdea9f0b0449657ad360ce5483e6924ecebeb4393ca4329442f61f5f9e5 Jul 09 02:00:11.798747 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.798679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" event={"ID":"13021762-a6ad-46ab-a4de-dc1838a6662e","Type":"ContainerStarted","Data":"7770b16811a1522a1710a77f23ec0da1f677f4b84f8e8f6e3c171067a63ed9e1"} Jul 09 02:00:11.800337 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.800310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv" event={"ID":"98d79bfc-4203-478c-bf3f-c6bac300e131","Type":"ContainerStarted","Data":"ac82fbdea9f0b0449657ad360ce5483e6924ecebeb4393ca4329442f61f5f9e5"} Jul 09 02:00:11.801664 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.801639 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" event={"ID":"71731d5b-3313-4046-8579-f828a13ba12c","Type":"ContainerStarted","Data":"4dbbff99107e8c4bd1ae14f3045da0e73dee02960c29cb3ea855cf504e97215e"} Jul 09 02:00:11.803497 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.803467 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2qgxg" event={"ID":"abb47d9b-697a-4099-8423-03edd6d52ad6","Type":"ContainerStarted","Data":"d1f30006dee3a94fe6c079ac6abb8d8866869bab6b5d845e4dfafdc968670302"} Jul 09 02:00:11.806709 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.806684 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerStarted","Data":"a3f21e1470e45b948c2db020192426635fa47ac73014e09eb73539c15b112989"} Jul 09 02:00:11.822211 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.822031 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" podStartSLOduration=9.315317528 podStartE2EDuration="14.822014251s" totalImagesPullingTime="5.506696723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:11.821523147 +0000 UTC m=+41.709685710" watchObservedRunningTime="2026-07-09 02:00:11.822014251 +0000 UTC m=+41.710176817" Jul 09 02:00:11.851456 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.851393 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2qgxg" podStartSLOduration=1.340678944 podStartE2EDuration="6.851371057s" totalImagesPullingTime="5.510692113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:11.849729727 +0000 UTC m=+41.737892293" watchObservedRunningTime="2026-07-09 02:00:11.851371057 +0000 UTC m=+41.739533624" Jul 09 02:00:11.912570 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.912523 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v"] Jul 09 02:00:11.915020 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:11.914991 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-4mkwb"] Jul 09 02:00:11.917217 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:11.917190 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8290af00_40ea_4657_bf4e_92137e8bc739.slice/crio-e22b32e8801263e2bcfc45fe7e36f1c1cd5c1c909e0d1310e1a00ee06a9eefad WatchSource:0}: Error finding container e22b32e8801263e2bcfc45fe7e36f1c1cd5c1c909e0d1310e1a00ee06a9eefad: Status 404 returned error can't find the container with id e22b32e8801263e2bcfc45fe7e36f1c1cd5c1c909e0d1310e1a00ee06a9eefad Jul 09 02:00:11.917920 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:11.917892 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84a9c83_d058_4182_814d_5e9a282d2013.slice/crio-c54bf3620bda5ccecdf635c0beff37a53411fa05c6452067ea25b4de15dac830 WatchSource:0}: Error finding container c54bf3620bda5ccecdf635c0beff37a53411fa05c6452067ea25b4de15dac830: Status 404 returned error can't find the container with id c54bf3620bda5ccecdf635c0beff37a53411fa05c6452067ea25b4de15dac830 Jul 09 02:00:12.668185 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.668097 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-msjzj"] Jul 09 02:00:12.689095 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.689064 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-msjzj"] Jul 09 02:00:12.689316 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.689203 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.691893 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.691865 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 02:00:12.692005 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.691869 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 02:00:12.692063 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.692015 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfnrl\"" Jul 09 02:00:12.728241 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.728200 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.728447 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.728339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/795ca69c-d52b-465e-a46d-454ced383b6b-data-volume\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.728447 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.728382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.728525 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.728488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28rq\" (UniqueName: \"kubernetes.io/projected/795ca69c-d52b-465e-a46d-454ced383b6b-kube-api-access-f28rq\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.728573 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.728539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795ca69c-d52b-465e-a46d-454ced383b6b-crio-socket\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.812676 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.812630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" event={"ID":"d84a9c83-d058-4182-814d-5e9a282d2013","Type":"ContainerStarted","Data":"c54bf3620bda5ccecdf635c0beff37a53411fa05c6452067ea25b4de15dac830"} Jul 09 02:00:12.814094 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.814064 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" event={"ID":"8290af00-40ea-4657-bf4e-92137e8bc739","Type":"ContainerStarted","Data":"e22b32e8801263e2bcfc45fe7e36f1c1cd5c1c909e0d1310e1a00ee06a9eefad"} Jul 09 02:00:12.817551 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.817518 2578 generic.go:356] "Generic (PLEG): container finished" podID="3c624a3e-0848-4b58-bb45-a1d1dbb59b8a" containerID="a3f21e1470e45b948c2db020192426635fa47ac73014e09eb73539c15b112989" exitCode=0 Jul 09 02:00:12.817693 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.817611 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerDied","Data":"a3f21e1470e45b948c2db020192426635fa47ac73014e09eb73539c15b112989"} Jul 09 02:00:12.829602 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.829570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.829765 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:12.829689 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 02:00:12.829765 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.829725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/795ca69c-d52b-465e-a46d-454ced383b6b-data-volume\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.829765 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:12.829756 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:13.329736 +0000 UTC m=+43.217898562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : secret "insights-runtime-extractor-tls" not found Jul 09 02:00:12.829933 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.829824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.829990 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.829941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28rq\" (UniqueName: \"kubernetes.io/projected/795ca69c-d52b-465e-a46d-454ced383b6b-kube-api-access-f28rq\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.830132 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.830005 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/795ca69c-d52b-465e-a46d-454ced383b6b-data-volume\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.830132 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.830013 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795ca69c-d52b-465e-a46d-454ced383b6b-crio-socket\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.830281 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:12.830150 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 02:00:12.830281 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:12.830199 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:13.330183091 +0000 UTC m=+43.218345653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : configmap "kube-rbac-proxy" not found Jul 09 02:00:12.835337 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.833492 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/795ca69c-d52b-465e-a46d-454ced383b6b-crio-socket\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:12.845164 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:12.845129 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28rq\" (UniqueName: \"kubernetes.io/projected/795ca69c-d52b-465e-a46d-454ced383b6b-kube-api-access-f28rq\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:13.338335 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.338301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:13.338515 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.338356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:13.338515 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.338392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:13.338515 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.338466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:13.338515 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.338495 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.338522 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338528 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338538 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338553 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b9d58d4f-zkvx7: secret "image-registry-tls" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338586 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338614 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:14.338592604 +0000 UTC m=+44.226755148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : secret "insights-runtime-extractor-tls" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338623 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.338548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338635 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338639 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:21.338622184 +0000 UTC m=+51.226784729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : secret "router-metrics-certs-default" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338685 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338690 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls podName:6d66a9ae-2d58-4260-a16d-096a4311a43a nodeName:}" failed. No retries permitted until 2026-07-09 02:00:21.338677926 +0000 UTC m=+51.226840469 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls") pod "image-registry-64b9d58d4f-zkvx7" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a") : secret "image-registry-tls" not found Jul 09 02:00:13.338726 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338713 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls podName:fb4e10fa-e512-499f-9cb3-9b98ed8bcba7 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:21.338704001 +0000 UTC m=+51.226866550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-72vjt" (UID: "fb4e10fa-e512-499f-9cb3-9b98ed8bcba7") : secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:13.339354 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338742 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:14.338734307 +0000 UTC m=+44.226896855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : configmap "kube-rbac-proxy" not found Jul 09 02:00:13.339354 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338756 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:21.338748204 +0000 UTC m=+51.226910747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : configmap references non-existent config key: service-ca.crt Jul 09 02:00:13.339354 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:13.338773 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls podName:50bc272c-2f83-44a1-9c62-6cbbd2e5a473 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:21.338765521 +0000 UTC m=+51.226928067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls") pod "dns-default-56q8v" (UID: "50bc272c-2f83-44a1-9c62-6cbbd2e5a473") : secret "dns-default-metrics-tls" not found Jul 09 02:00:13.823932 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.823817 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv" event={"ID":"98d79bfc-4203-478c-bf3f-c6bac300e131","Type":"ContainerStarted","Data":"9dcabaaa14765ea452357ee29914b9f59289c2c0c8e03b0c4df01d19c72c5f0e"} Jul 09 02:00:13.828141 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.827982 2578 generic.go:356] "Generic (PLEG): container finished" podID="3c624a3e-0848-4b58-bb45-a1d1dbb59b8a" containerID="a1cda15c70ed3d5e755995729d43eb84de8185655ff8a46dcf3ed8e176b063ea" exitCode=0 Jul 09 02:00:13.828141 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.828055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerDied","Data":"a1cda15c70ed3d5e755995729d43eb84de8185655ff8a46dcf3ed8e176b063ea"} Jul 09 02:00:13.842049 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:13.841991 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-dg9nv" podStartSLOduration=5.062923192 podStartE2EDuration="6.841967909s" totalImagesPullingTime="1.779044717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:13.840346299 +0000 UTC m=+43.728508877" watchObservedRunningTime="2026-07-09 02:00:13.841967909 +0000 UTC m=+43.730130507" Jul 09 02:00:14.263944 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:14.263920 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hg8mp_e1f20951-58ef-4818-8715-c74ff5029e4d/dns-node-resolver/0.log" Jul 09 02:00:14.349477 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:14.349446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:14.349660 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:14.349569 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:14.349660 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:14.349595 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 02:00:14.349764 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:14.349675 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 02:00:14.349764 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:14.349685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:16.349663197 +0000 UTC m=+46.237825746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : secret "insights-runtime-extractor-tls" not found Jul 09 02:00:14.349764 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:14.349727 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:16.349711598 +0000 UTC m=+46.237874141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : configmap "kube-rbac-proxy" not found Jul 09 02:00:14.835488 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:14.835451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" event={"ID":"3c624a3e-0848-4b58-bb45-a1d1dbb59b8a","Type":"ContainerStarted","Data":"7b07446d73182d31091e474fa26a623a72e8d18b14fc12b3517bdde33e55e62a"} Jul 09 02:00:14.861930 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:14.861874 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jqxdt" podStartSLOduration=8.455101323 podStartE2EDuration="43.861855801s" totalImagesPullingTime="35.406754478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:14.860350465 +0000 UTC m=+44.748513031" watchObservedRunningTime="2026-07-09 02:00:14.861855801 +0000 UTC m=+44.750018365" Jul 09 02:00:15.264216 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:15.264189 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w4w95_70db81f0-b646-4ae3-ab59-02cb9bc52828/node-ca/0.log" Jul 09 02:00:15.462514 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:15.462471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:15.462691 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:15.462635 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 02:00:15.462739 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:15.462719 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls podName:a3cca3f2-372d-47d6-9d00-a21c7e187716 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:23.462695066 +0000 UTC m=+53.350857686 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-94cb7" (UID: "a3cca3f2-372d-47d6-9d00-a21c7e187716") : secret "samples-operator-tls" not found Jul 09 02:00:15.663509 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:15.663424 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2qgxg_abb47d9b-697a-4099-8423-03edd6d52ad6/serve-healthcheck-canary/0.log" Jul 09 02:00:16.372663 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.372628 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:16.373122 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.372724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:16.373122 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:16.372814 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 02:00:16.373122 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:16.372854 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 02:00:16.373122 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:16.372897 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:20.372873068 +0000 UTC m=+50.261035631 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : secret "insights-runtime-extractor-tls" not found Jul 09 02:00:16.373122 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:16.372924 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:20.372904834 +0000 UTC m=+50.261067381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : configmap "kube-rbac-proxy" not found Jul 09 02:00:16.843201 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.843171 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/0.log" Jul 09 02:00:16.843413 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.843211 2578 generic.go:356] "Generic (PLEG): container finished" podID="d84a9c83-d058-4182-814d-5e9a282d2013" containerID="42c16212b27501f5e8366c102e61fd34564e62f6dd67b9d38e2f09043fef6f61" exitCode=255 Jul 09 02:00:16.843413 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.843314 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" event={"ID":"d84a9c83-d058-4182-814d-5e9a282d2013","Type":"ContainerDied","Data":"42c16212b27501f5e8366c102e61fd34564e62f6dd67b9d38e2f09043fef6f61"} Jul 09 02:00:16.843739 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.843718 2578 scope.go:117] "RemoveContainer" containerID="42c16212b27501f5e8366c102e61fd34564e62f6dd67b9d38e2f09043fef6f61" Jul 09 02:00:16.845302 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.845275 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" event={"ID":"8290af00-40ea-4657-bf4e-92137e8bc739","Type":"ContainerStarted","Data":"dae972878f351922c4f95cb8e36dc2677e6e2171a43dcae085b6ca59ff143c8d"} Jul 09 02:00:16.846985 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.846957 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" event={"ID":"71731d5b-3313-4046-8579-f828a13ba12c","Type":"ContainerStarted","Data":"1d4230af9a077acae1bb0413eec711e0b09f50437f0c3bdfca54ee0cd44fb456"} Jul 09 02:00:16.876216 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:16.876162 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" podStartSLOduration=5.627175994 podStartE2EDuration="9.87614133s" totalImagesPullingTime="4.248965336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:16.875944166 +0000 UTC m=+46.764106730" watchObservedRunningTime="2026-07-09 02:00:16.87614133 +0000 UTC m=+46.764303876" Jul 09 02:00:17.852100 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:17.852069 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:00:17.852525 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:17.852464 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/0.log" Jul 09 02:00:17.852525 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:17.852499 2578 generic.go:356] "Generic (PLEG): container finished" podID="d84a9c83-d058-4182-814d-5e9a282d2013" containerID="422bf454d7c8fdf447451762bc821b85ea9ea6aecb1fb5f4ea59adf8c6ae8506" exitCode=255 Jul 09 02:00:17.852648 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:17.852584 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" event={"ID":"d84a9c83-d058-4182-814d-5e9a282d2013","Type":"ContainerDied","Data":"422bf454d7c8fdf447451762bc821b85ea9ea6aecb1fb5f4ea59adf8c6ae8506"} Jul 09 02:00:17.852648 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:17.852638 2578 scope.go:117] "RemoveContainer" containerID="42c16212b27501f5e8366c102e61fd34564e62f6dd67b9d38e2f09043fef6f61" Jul 09 02:00:17.852889 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:17.852872 2578 scope.go:117] "RemoveContainer" containerID="422bf454d7c8fdf447451762bc821b85ea9ea6aecb1fb5f4ea59adf8c6ae8506" Jul 09 02:00:17.853082 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:17.853062 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-4mkwb_openshift-console-operator(d84a9c83-d058-4182-814d-5e9a282d2013)\"" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" podUID="d84a9c83-d058-4182-814d-5e9a282d2013" Jul 09 02:00:17.876087 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:17.876045 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" podStartSLOduration=6.40951322 podStartE2EDuration="10.876029455s" totalImagesPullingTime="4.466516235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:16.89384442 +0000 UTC m=+46.782006986" watchObservedRunningTime="2026-07-09 02:00:17.876029455 +0000 UTC m=+47.764192096" Jul 09 02:00:18.081216 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:18.081173 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:18.081396 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:18.081233 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:18.856690 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:18.856661 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:00:18.857159 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:18.857140 2578 scope.go:117] "RemoveContainer" containerID="422bf454d7c8fdf447451762bc821b85ea9ea6aecb1fb5f4ea59adf8c6ae8506" Jul 09 02:00:18.857389 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:18.857364 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-4mkwb_openshift-console-operator(d84a9c83-d058-4182-814d-5e9a282d2013)\"" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" podUID="d84a9c83-d058-4182-814d-5e9a282d2013" Jul 09 02:00:19.860141 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:19.860111 2578 scope.go:117] "RemoveContainer" containerID="422bf454d7c8fdf447451762bc821b85ea9ea6aecb1fb5f4ea59adf8c6ae8506" Jul 09 02:00:19.860563 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:19.860293 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-4mkwb_openshift-console-operator(d84a9c83-d058-4182-814d-5e9a282d2013)\"" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" podUID="d84a9c83-d058-4182-814d-5e9a282d2013" Jul 09 02:00:20.409894 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:20.409856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:20.410123 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:20.409961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:20.410123 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:20.410029 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 02:00:20.410123 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:20.410075 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 02:00:20.410123 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:20.410113 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:28.410090529 +0000 UTC m=+58.298253074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : secret "insights-runtime-extractor-tls" not found Jul 09 02:00:20.410382 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:20.410135 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:28.410124805 +0000 UTC m=+58.298287350 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : configmap "kube-rbac-proxy" not found Jul 09 02:00:21.419701 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:21.419657 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:21.419713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.419802 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.419815 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.419838 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b9d58d4f-zkvx7: secret "image-registry-tls" not found Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.419855 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:37.419840454 +0000 UTC m=+67.308003000 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : secret "router-metrics-certs-default" not found Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.419881 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls podName:6d66a9ae-2d58-4260-a16d-096a4311a43a nodeName:}" failed. No retries permitted until 2026-07-09 02:00:37.419869563 +0000 UTC m=+67.308032106 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls") pod "image-registry-64b9d58d4f-zkvx7" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a") : secret "image-registry-tls" not found Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:21.419917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:21.419953 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:21.419971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.420055 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.420070 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle podName:a98ef70b-a070-45fd-910e-0353efdd0278 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:37.42005844 +0000 UTC m=+67.308221005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle") pod "router-default-789b84dcd9-wrbcr" (UID: "a98ef70b-a070-45fd-910e-0353efdd0278") : configmap references non-existent config key: service-ca.crt Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.420097 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls podName:50bc272c-2f83-44a1-9c62-6cbbd2e5a473 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:37.420084515 +0000 UTC m=+67.308247071 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls") pod "dns-default-56q8v" (UID: "50bc272c-2f83-44a1-9c62-6cbbd2e5a473") : secret "dns-default-metrics-tls" not found Jul 09 02:00:21.420105 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.420109 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:21.420563 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:21.420147 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls podName:fb4e10fa-e512-499f-9cb3-9b98ed8bcba7 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:37.420138798 +0000 UTC m=+67.308301342 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-72vjt" (UID: "fb4e10fa-e512-499f-9cb3-9b98ed8bcba7") : secret "cluster-monitoring-operator-tls" not found Jul 09 02:00:23.541973 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:23.541932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:23.542459 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:23.542052 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 02:00:23.542459 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:23.542108 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls podName:a3cca3f2-372d-47d6-9d00-a21c7e187716 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:39.542094218 +0000 UTC m=+69.430256761 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-94cb7" (UID: "a3cca3f2-372d-47d6-9d00-a21c7e187716") : secret "samples-operator-tls" not found Jul 09 02:00:28.487647 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:28.487609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:28.488033 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:28.487712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:28.488033 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:28.487761 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 02:00:28.488033 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:28.487828 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls podName:795ca69c-d52b-465e-a46d-454ced383b6b nodeName:}" failed. No retries permitted until 2026-07-09 02:00:44.487812484 +0000 UTC m=+74.375975043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-msjzj" (UID: "795ca69c-d52b-465e-a46d-454ced383b6b") : secret "insights-runtime-extractor-tls" not found Jul 09 02:00:28.488241 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:28.488221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/795ca69c-d52b-465e-a46d-454ced383b6b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:30.773848 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:30.773816 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d7cp8" Jul 09 02:00:34.575265 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:34.575226 2578 scope.go:117] "RemoveContainer" containerID="422bf454d7c8fdf447451762bc821b85ea9ea6aecb1fb5f4ea59adf8c6ae8506" Jul 09 02:00:34.905334 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:34.905240 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:00:34.905471 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:34.905333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" event={"ID":"d84a9c83-d058-4182-814d-5e9a282d2013","Type":"ContainerStarted","Data":"d318b98222a1f248db3ae7f072b780815485cd275d2cbbf5c60c086937abc0a1"} Jul 09 02:00:34.905618 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:34.905600 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:34.927895 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:34.927839 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" podStartSLOduration=23.674675002 podStartE2EDuration="27.927823519s" totalImagesPullingTime="4.253148517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:34.927087622 +0000 UTC m=+64.815250193" watchObservedRunningTime="2026-07-09 02:00:34.927823519 +0000 UTC m=+64.815986083" Jul 09 02:00:35.184978 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:35.184906 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-4mkwb" Jul 09 02:00:36.356966 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.356926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 02:00:36.356966 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.356970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 02:00:36.359970 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.359950 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 02:00:36.360124 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.360106 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 02:00:36.369792 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.369760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2f27e32-08c7-4607-9805-e280bcbcd8fe-original-pull-secret\") pod \"global-pull-secret-syncer-fg9k4\" (UID: \"c2f27e32-08c7-4607-9805-e280bcbcd8fe\") " pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 02:00:36.369792 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.369781 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c847f9eb-ebc3-40a4-a78c-e14cca47a814-metrics-certs\") pod \"network-metrics-daemon-t7q22\" (UID: \"c847f9eb-ebc3-40a4-a78c-e14cca47a814\") " pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 02:00:36.457475 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.457433 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:36.460317 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.460298 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 02:00:36.471353 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.471327 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 02:00:36.481155 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.481130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw9l7\" (UniqueName: \"kubernetes.io/projected/aaa28b10-cbbd-463c-a8a5-e5137a4070f6-kube-api-access-lw9l7\") pod \"network-check-target-jzpk6\" (UID: \"aaa28b10-cbbd-463c-a8a5-e5137a4070f6\") " pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:36.483864 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.483841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fg9k4" Jul 09 02:00:36.492187 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.492166 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmqn4\"" Jul 09 02:00:36.496877 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.496854 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j65lj\"" Jul 09 02:00:36.499999 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.499980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t7q22" Jul 09 02:00:36.505263 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.505230 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:36.637782 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.637699 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fg9k4"] Jul 09 02:00:36.644104 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:36.644070 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f27e32_08c7_4607_9805_e280bcbcd8fe.slice/crio-3302bbe473390fcde0bec8251afa8ff9563187adfc4e0f1aaf7f3111d3c6fda0 WatchSource:0}: Error finding container 3302bbe473390fcde0bec8251afa8ff9563187adfc4e0f1aaf7f3111d3c6fda0: Status 404 returned error can't find the container with id 3302bbe473390fcde0bec8251afa8ff9563187adfc4e0f1aaf7f3111d3c6fda0 Jul 09 02:00:36.659475 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.659433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t7q22"] Jul 09 02:00:36.662242 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:36.662210 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc847f9eb_ebc3_40a4_a78c_e14cca47a814.slice/crio-8686b8334615a9d39a8e68e0fa3f7cb92088733174e099eb00114bd894040142 WatchSource:0}: Error finding container 8686b8334615a9d39a8e68e0fa3f7cb92088733174e099eb00114bd894040142: Status 404 returned error can't find the container with id 8686b8334615a9d39a8e68e0fa3f7cb92088733174e099eb00114bd894040142 Jul 09 02:00:36.673025 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.672970 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jzpk6"] Jul 09 02:00:36.675355 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:36.675330 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa28b10_cbbd_463c_a8a5_e5137a4070f6.slice/crio-4dea49eb77b498fab65c900c5cb4c585c28d6344b4e1d3a85234d102a833eb08 WatchSource:0}: Error finding container 4dea49eb77b498fab65c900c5cb4c585c28d6344b4e1d3a85234d102a833eb08: Status 404 returned error can't find the container with id 4dea49eb77b498fab65c900c5cb4c585c28d6344b4e1d3a85234d102a833eb08 Jul 09 02:00:36.912697 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.912608 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jzpk6" event={"ID":"aaa28b10-cbbd-463c-a8a5-e5137a4070f6","Type":"ContainerStarted","Data":"4dea49eb77b498fab65c900c5cb4c585c28d6344b4e1d3a85234d102a833eb08"} Jul 09 02:00:36.913715 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.913689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fg9k4" event={"ID":"c2f27e32-08c7-4607-9805-e280bcbcd8fe","Type":"ContainerStarted","Data":"3302bbe473390fcde0bec8251afa8ff9563187adfc4e0f1aaf7f3111d3c6fda0"} Jul 09 02:00:36.914776 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:36.914747 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t7q22" event={"ID":"c847f9eb-ebc3-40a4-a78c-e14cca47a814","Type":"ContainerStarted","Data":"8686b8334615a9d39a8e68e0fa3f7cb92088733174e099eb00114bd894040142"} Jul 09 02:00:37.468178 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.468119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:37.468178 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.468175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:37.468748 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.468208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:37.468748 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.468324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:37.468748 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.468374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:37.469007 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.468978 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a98ef70b-a070-45fd-910e-0353efdd0278-service-ca-bundle\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:37.471346 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.471294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/50bc272c-2f83-44a1-9c62-6cbbd2e5a473-metrics-tls\") pod \"dns-default-56q8v\" (UID: \"50bc272c-2f83-44a1-9c62-6cbbd2e5a473\") " pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:37.472018 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.471539 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/fb4e10fa-e512-499f-9cb3-9b98ed8bcba7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-72vjt\" (UID: \"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:37.472018 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.471978 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a98ef70b-a070-45fd-910e-0353efdd0278-metrics-certs\") pod \"router-default-789b84dcd9-wrbcr\" (UID: \"a98ef70b-a070-45fd-910e-0353efdd0278\") " pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:37.474640 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.474615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"image-registry-64b9d58d4f-zkvx7\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:37.609216 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.609180 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hmfxw\"" Jul 09 02:00:37.616832 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.616800 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:37.625419 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.625389 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-flcfn\"" Jul 09 02:00:37.634451 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.634424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" Jul 09 02:00:37.637787 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.637752 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nnpdv\"" Jul 09 02:00:37.644186 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.644129 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8nxtt\"" Jul 09 02:00:37.645476 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.645449 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:37.650732 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.650317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:37.824607 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.824493 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64b9d58d4f-zkvx7"] Jul 09 02:00:37.874558 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.874398 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt"] Jul 09 02:00:37.880815 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.880595 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-56q8v"] Jul 09 02:00:37.913952 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:37.913923 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-789b84dcd9-wrbcr"] Jul 09 02:00:38.083494 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:38.083408 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d66a9ae_2d58_4260_a16d_096a4311a43a.slice/crio-8f640153884aeea7a95f430c1c0174f193a22817e8f71ccbae41c2387be5d49f WatchSource:0}: Error finding container 8f640153884aeea7a95f430c1c0174f193a22817e8f71ccbae41c2387be5d49f: Status 404 returned error can't find the container with id 8f640153884aeea7a95f430c1c0174f193a22817e8f71ccbae41c2387be5d49f Jul 09 02:00:38.084157 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:38.084129 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4e10fa_e512_499f_9cb3_9b98ed8bcba7.slice/crio-b99540a893ea119bce78817c2a06e6e86d4c7d931f633dc5eda30c0d3e749f35 WatchSource:0}: Error finding container b99540a893ea119bce78817c2a06e6e86d4c7d931f633dc5eda30c0d3e749f35: Status 404 returned error can't find the container with id b99540a893ea119bce78817c2a06e6e86d4c7d931f633dc5eda30c0d3e749f35 Jul 09 02:00:38.926262 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.926155 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" event={"ID":"6d66a9ae-2d58-4260-a16d-096a4311a43a","Type":"ContainerStarted","Data":"4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866"} Jul 09 02:00:38.926262 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.926221 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" event={"ID":"6d66a9ae-2d58-4260-a16d-096a4311a43a","Type":"ContainerStarted","Data":"8f640153884aeea7a95f430c1c0174f193a22817e8f71ccbae41c2387be5d49f"} Jul 09 02:00:38.926986 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.926936 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:00:38.930435 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.930404 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t7q22" event={"ID":"c847f9eb-ebc3-40a4-a78c-e14cca47a814","Type":"ContainerStarted","Data":"05a3d74de040e826536cda93a6cab08b4e24f5d66bb349611f92fe47d6f64e23"} Jul 09 02:00:38.930567 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.930462 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t7q22" event={"ID":"c847f9eb-ebc3-40a4-a78c-e14cca47a814","Type":"ContainerStarted","Data":"e1a3b6c71a657e2628d3a357c518f42c0b7d96a4d6718cb983ce8d784332c51a"} Jul 09 02:00:38.933630 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.933572 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" event={"ID":"a98ef70b-a070-45fd-910e-0353efdd0278","Type":"ContainerStarted","Data":"5cfecddb7a1b9839abeeae9f0533c6232f2e0368da6c93e0cb875e7ed5ccd560"} Jul 09 02:00:38.933758 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.933650 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" event={"ID":"a98ef70b-a070-45fd-910e-0353efdd0278","Type":"ContainerStarted","Data":"a991f93bd5bfc11a88a70387b08084ee1461630d0ac11a89839de7fdf781d677"} Jul 09 02:00:38.935636 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.935587 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" event={"ID":"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7","Type":"ContainerStarted","Data":"b99540a893ea119bce78817c2a06e6e86d4c7d931f633dc5eda30c0d3e749f35"} Jul 09 02:00:38.937079 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.937015 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-56q8v" event={"ID":"50bc272c-2f83-44a1-9c62-6cbbd2e5a473","Type":"ContainerStarted","Data":"fe08cdd0d88af481f5e92a060095159c7e1adddfe66e3a5dbf1f95264a7b6e06"} Jul 09 02:00:38.957569 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.956722 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" podStartSLOduration=67.956699665 podStartE2EDuration="1m7.956699665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:38.95465733 +0000 UTC m=+68.842819899" watchObservedRunningTime="2026-07-09 02:00:38.956699665 +0000 UTC m=+68.844862232" Jul 09 02:00:38.981931 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:38.981862 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" podStartSLOduration=41.9818409 podStartE2EDuration="41.9818409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:38.981631974 +0000 UTC m=+68.869794541" watchObservedRunningTime="2026-07-09 02:00:38.9818409 +0000 UTC m=+68.870003471" Jul 09 02:00:39.012417 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.011513 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-t7q22" podStartSLOduration=66.534150852 podStartE2EDuration="1m8.011492238s" totalImagesPullingTime="1.477341386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:39.011466167 +0000 UTC m=+68.899628732" watchObservedRunningTime="2026-07-09 02:00:39.011492238 +0000 UTC m=+68.899654804" Jul 09 02:00:39.591897 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.591792 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:39.602881 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.602847 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/a3cca3f2-372d-47d6-9d00-a21c7e187716-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-94cb7\" (UID: \"a3cca3f2-372d-47d6-9d00-a21c7e187716\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:39.646725 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.646690 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:39.649871 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.649844 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:39.740885 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.740824 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nn8t6\"" Jul 09 02:00:39.748571 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.748526 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" Jul 09 02:00:39.942108 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.941995 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:39.943297 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:39.943274 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-789b84dcd9-wrbcr" Jul 09 02:00:43.068718 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.068689 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7"] Jul 09 02:00:43.956038 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.955995 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" event={"ID":"fb4e10fa-e512-499f-9cb3-9b98ed8bcba7","Type":"ContainerStarted","Data":"32f311c4476f8530ab59f17c9ece8827e2f752f7f28a371636fe6aa2e53a4626"} Jul 09 02:00:43.961060 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.961024 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jzpk6" event={"ID":"aaa28b10-cbbd-463c-a8a5-e5137a4070f6","Type":"ContainerStarted","Data":"bfa48912a192b3e04d119f74bce72a626eceb83f27241fee34237cd85429196a"} Jul 09 02:00:43.961232 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.961081 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:00:43.962473 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.962453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fg9k4" event={"ID":"c2f27e32-08c7-4607-9805-e280bcbcd8fe","Type":"ContainerStarted","Data":"dae42836b00854d712f4471a0b662e5ab1fcc24b7596f544325c3bc2e3525faa"} Jul 09 02:00:43.964234 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.964211 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-56q8v" event={"ID":"50bc272c-2f83-44a1-9c62-6cbbd2e5a473","Type":"ContainerStarted","Data":"432e926103da99abe90640a5e646ddc14a17b6d06dbf87fafb62b76ba351cde2"} Jul 09 02:00:43.964345 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.964264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-56q8v" event={"ID":"50bc272c-2f83-44a1-9c62-6cbbd2e5a473","Type":"ContainerStarted","Data":"d3162290390d423a4cefecf44b3056a513f4f46782d84c12c693e350651b1606"} Jul 09 02:00:43.964419 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.964352 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:43.965262 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.965233 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" event={"ID":"a3cca3f2-372d-47d6-9d00-a21c7e187716","Type":"ContainerStarted","Data":"e8ecb4002bfb275eff88f1806d649548800286a9ce31dfcdeb7b2a9a30739d6a"} Jul 09 02:00:43.974654 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.974608 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-72vjt" podStartSLOduration=42.131545961 podStartE2EDuration="46.974591821s" totalImagesPullingTime="4.84304586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:43.974052827 +0000 UTC m=+73.862215402" watchObservedRunningTime="2026-07-09 02:00:43.974591821 +0000 UTC m=+73.862754428" Jul 09 02:00:43.991229 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:43.991178 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jzpk6" podStartSLOduration=66.739922663 podStartE2EDuration="1m12.991162925s" totalImagesPullingTime="6.251240262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:43.990298706 +0000 UTC m=+73.878461271" watchObservedRunningTime="2026-07-09 02:00:43.991162925 +0000 UTC m=+73.879325489" Jul 09 02:00:44.005881 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.005832 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fg9k4" podStartSLOduration=66.407449886 podStartE2EDuration="1m13.00581652s" totalImagesPullingTime="6.598366634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 01:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:44.005004808 +0000 UTC m=+73.893167372" watchObservedRunningTime="2026-07-09 02:00:44.00581652 +0000 UTC m=+73.893979086" Jul 09 02:00:44.025556 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.025499 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-56q8v" podStartSLOduration=34.179148793 podStartE2EDuration="39.025481494s" totalImagesPullingTime="4.846332701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:44.024777034 +0000 UTC m=+73.912939595" watchObservedRunningTime="2026-07-09 02:00:44.025481494 +0000 UTC m=+73.913644058" Jul 09 02:00:44.321710 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.321664 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr"] Jul 09 02:00:44.349295 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.349227 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-9cmn7"] Jul 09 02:00:44.349480 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.349383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" Jul 09 02:00:44.352712 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.352519 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 02:00:44.352712 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.352639 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6247f\"" Jul 09 02:00:44.352898 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.352717 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 02:00:44.365370 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.365348 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr"] Jul 09 02:00:44.365512 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.365389 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9cmn7"] Jul 09 02:00:44.365512 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.365459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9cmn7" Jul 09 02:00:44.371216 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.371193 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 02:00:44.371345 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.371273 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kdjt9\"" Jul 09 02:00:44.371885 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.371863 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 02:00:44.533761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.533677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/369d323f-66e4-4b9a-a354-35926a34dfc3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rtsbr\" (UID: \"369d323f-66e4-4b9a-a354-35926a34dfc3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" Jul 09 02:00:44.533761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.533715 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkp6\" (UniqueName: \"kubernetes.io/projected/92d4c717-baf8-4e62-b139-3e67293e89d5-kube-api-access-7kkp6\") pod \"downloads-685bfb5f64-9cmn7\" (UID: \"92d4c717-baf8-4e62-b139-3e67293e89d5\") " pod="openshift-console/downloads-685bfb5f64-9cmn7" Jul 09 02:00:44.533951 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.533847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/369d323f-66e4-4b9a-a354-35926a34dfc3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rtsbr\" (UID: \"369d323f-66e4-4b9a-a354-35926a34dfc3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" Jul 09 02:00:44.533951 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.533889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:44.536652 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.536623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/795ca69c-d52b-465e-a46d-454ced383b6b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-msjzj\" (UID: \"795ca69c-d52b-465e-a46d-454ced383b6b\") " pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:44.634893 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.634856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/369d323f-66e4-4b9a-a354-35926a34dfc3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rtsbr\" (UID: \"369d323f-66e4-4b9a-a354-35926a34dfc3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" Jul 09 02:00:44.635071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.634913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/369d323f-66e4-4b9a-a354-35926a34dfc3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rtsbr\" (UID: \"369d323f-66e4-4b9a-a354-35926a34dfc3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" Jul 09 02:00:44.635071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.634935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkp6\" (UniqueName: \"kubernetes.io/projected/92d4c717-baf8-4e62-b139-3e67293e89d5-kube-api-access-7kkp6\") pod \"downloads-685bfb5f64-9cmn7\" (UID: \"92d4c717-baf8-4e62-b139-3e67293e89d5\") " pod="openshift-console/downloads-685bfb5f64-9cmn7" Jul 09 02:00:44.635649 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.635627 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/369d323f-66e4-4b9a-a354-35926a34dfc3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rtsbr\" (UID: \"369d323f-66e4-4b9a-a354-35926a34dfc3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" Jul 09 02:00:44.637380 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.637360 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/369d323f-66e4-4b9a-a354-35926a34dfc3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rtsbr\" (UID: \"369d323f-66e4-4b9a-a354-35926a34dfc3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" Jul 09 02:00:44.661525 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.661491 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" Jul 09 02:00:44.699602 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.699575 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkp6\" (UniqueName: \"kubernetes.io/projected/92d4c717-baf8-4e62-b139-3e67293e89d5-kube-api-access-7kkp6\") pod \"downloads-685bfb5f64-9cmn7\" (UID: \"92d4c717-baf8-4e62-b139-3e67293e89d5\") " pod="openshift-console/downloads-685bfb5f64-9cmn7" Jul 09 02:00:44.801828 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.801736 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vfnrl\"" Jul 09 02:00:44.809488 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.809456 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-msjzj" Jul 09 02:00:44.824989 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.824959 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr"] Jul 09 02:00:44.828175 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:44.828126 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod369d323f_66e4_4b9a_a354_35926a34dfc3.slice/crio-950eac328631646277b6eb2c0638874832798851d4fe92935a7d9bd0c6d51273 WatchSource:0}: Error finding container 950eac328631646277b6eb2c0638874832798851d4fe92935a7d9bd0c6d51273: Status 404 returned error can't find the container with id 950eac328631646277b6eb2c0638874832798851d4fe92935a7d9bd0c6d51273 Jul 09 02:00:44.970845 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.970811 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" event={"ID":"369d323f-66e4-4b9a-a354-35926a34dfc3","Type":"ContainerStarted","Data":"950eac328631646277b6eb2c0638874832798851d4fe92935a7d9bd0c6d51273"} Jul 09 02:00:44.976492 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.976468 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-9cmn7" Jul 09 02:00:44.980611 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:44.980593 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-msjzj"] Jul 09 02:00:44.982991 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:44.982966 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795ca69c_d52b_465e_a46d_454ced383b6b.slice/crio-41dd45ab5e0045b71a3f8c7995f34941410b66898f4454438b8061a712f17291 WatchSource:0}: Error finding container 41dd45ab5e0045b71a3f8c7995f34941410b66898f4454438b8061a712f17291: Status 404 returned error can't find the container with id 41dd45ab5e0045b71a3f8c7995f34941410b66898f4454438b8061a712f17291 Jul 09 02:00:45.143184 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.143152 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-9cmn7"] Jul 09 02:00:45.330994 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.330901 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5488b54ffc-fwq8d"] Jul 09 02:00:45.368188 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.368146 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5488b54ffc-fwq8d"] Jul 09 02:00:45.368352 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.368308 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.372805 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.372770 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 02:00:45.373116 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.373098 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ldhq7\"" Jul 09 02:00:45.373207 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.373098 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 02:00:45.373363 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.373342 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 02:00:45.373921 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.373900 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 02:00:45.374008 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.373908 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 02:00:45.541820 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.541784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-oauth-config\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.542018 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.541875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-serving-cert\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.542018 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.541959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-oauth-serving-cert\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.542018 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.542011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-console-config\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.542177 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.542114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-service-ca\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.542177 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.542149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xs47\" (UniqueName: \"kubernetes.io/projected/54465bc4-c858-45a7-820a-656af3984054-kube-api-access-8xs47\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.638518 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:45.638438 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d4c717_baf8_4e62_b139_3e67293e89d5.slice/crio-0aac347dbc5baaefe6f2c95195bbb28f14ab9e89cce831aac7996dd509d71392 WatchSource:0}: Error finding container 0aac347dbc5baaefe6f2c95195bbb28f14ab9e89cce831aac7996dd509d71392: Status 404 returned error can't find the container with id 0aac347dbc5baaefe6f2c95195bbb28f14ab9e89cce831aac7996dd509d71392 Jul 09 02:00:45.643387 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.643360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-oauth-serving-cert\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.643476 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.643418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-console-config\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.643476 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.643469 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-service-ca\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.643593 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.643503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xs47\" (UniqueName: \"kubernetes.io/projected/54465bc4-c858-45a7-820a-656af3984054-kube-api-access-8xs47\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.643593 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.643554 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-oauth-config\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.643687 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.643601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-serving-cert\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.644157 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.644123 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-console-config\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.644260 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.644127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-oauth-serving-cert\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.644388 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.644370 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-service-ca\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.646044 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.646022 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-serving-cert\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.646129 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.646111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-oauth-config\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.653498 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.653463 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xs47\" (UniqueName: \"kubernetes.io/projected/54465bc4-c858-45a7-820a-656af3984054-kube-api-access-8xs47\") pod \"console-5488b54ffc-fwq8d\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.679348 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.679312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:45.821808 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.821784 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5488b54ffc-fwq8d"] Jul 09 02:00:45.968227 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:00:45.968193 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54465bc4_c858_45a7_820a_656af3984054.slice/crio-f057f87801fbe8aae714bbbea296db896569d970f06cc4fce635688880eb6127 WatchSource:0}: Error finding container f057f87801fbe8aae714bbbea296db896569d970f06cc4fce635688880eb6127: Status 404 returned error can't find the container with id f057f87801fbe8aae714bbbea296db896569d970f06cc4fce635688880eb6127 Jul 09 02:00:45.975398 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.975370 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5488b54ffc-fwq8d" event={"ID":"54465bc4-c858-45a7-820a-656af3984054","Type":"ContainerStarted","Data":"f057f87801fbe8aae714bbbea296db896569d970f06cc4fce635688880eb6127"} Jul 09 02:00:45.976415 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.976390 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9cmn7" event={"ID":"92d4c717-baf8-4e62-b139-3e67293e89d5","Type":"ContainerStarted","Data":"0aac347dbc5baaefe6f2c95195bbb28f14ab9e89cce831aac7996dd509d71392"} Jul 09 02:00:45.977518 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:45.977498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-msjzj" event={"ID":"795ca69c-d52b-465e-a46d-454ced383b6b","Type":"ContainerStarted","Data":"41dd45ab5e0045b71a3f8c7995f34941410b66898f4454438b8061a712f17291"} Jul 09 02:00:46.985174 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:46.984930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" event={"ID":"a3cca3f2-372d-47d6-9d00-a21c7e187716","Type":"ContainerStarted","Data":"319af413d193433255f8697aa81082a925d6e86250c56515a8bb7374e0cb2c79"} Jul 09 02:00:46.985174 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:46.985052 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" event={"ID":"a3cca3f2-372d-47d6-9d00-a21c7e187716","Type":"ContainerStarted","Data":"1e6dcfd74bc877f3149f10d62b3c55e6a72950778f03ebd128dc91571f1f61d9"} Jul 09 02:00:47.005824 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:47.005501 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-94cb7" podStartSLOduration=37.246254408 podStartE2EDuration="40.005478373s" totalImagesPullingTime="2.759223965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:47.005295009 +0000 UTC m=+76.893457575" watchObservedRunningTime="2026-07-09 02:00:47.005478373 +0000 UTC m=+76.893640938" Jul 09 02:00:47.994052 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:47.993717 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" event={"ID":"369d323f-66e4-4b9a-a354-35926a34dfc3","Type":"ContainerStarted","Data":"9312f0656e28e3e8071fe686fe6e95646f978a409f6b7a534871c64ceb17ce35"} Jul 09 02:00:48.012002 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:48.011345 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rtsbr" podStartSLOduration=1.329015185 podStartE2EDuration="4.011325035s" totalImagesPullingTime="2.68230985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:48.010092892 +0000 UTC m=+77.898255460" watchObservedRunningTime="2026-07-09 02:00:48.011325035 +0000 UTC m=+77.899487659" Jul 09 02:00:52.013350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:52.013309 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-msjzj" event={"ID":"795ca69c-d52b-465e-a46d-454ced383b6b","Type":"ContainerStarted","Data":"ef39a4e2830020c6ed36c6de19f58f605f39844c2b2c7052d7c686c41b1ea949"} Jul 09 02:00:52.015159 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:52.015122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5488b54ffc-fwq8d" event={"ID":"54465bc4-c858-45a7-820a-656af3984054","Type":"ContainerStarted","Data":"f2f3a7a63b213d5b2a31c2915bfe6d5f326d49f0408662db8a4f59aeae04286a"} Jul 09 02:00:52.033109 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:52.033009 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5488b54ffc-fwq8d" podStartSLOduration=1.235719014 podStartE2EDuration="7.032992901s" totalImagesPullingTime="5.797273887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:52.031848039 +0000 UTC m=+81.920010606" watchObservedRunningTime="2026-07-09 02:00:52.032992901 +0000 UTC m=+81.921155466" Jul 09 02:00:53.021319 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:53.021279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-msjzj" event={"ID":"795ca69c-d52b-465e-a46d-454ced383b6b","Type":"ContainerStarted","Data":"8efda00a4ea89554d569c4108ffcf925785a90ef4d9493a25b95e557f8740f91"} Jul 09 02:00:53.974430 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:53.974181 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-56q8v" Jul 09 02:00:55.679800 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:55.679758 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:55.680277 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:55.679815 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:55.687739 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:55.687703 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:56.039546 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:56.039424 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-msjzj" event={"ID":"795ca69c-d52b-465e-a46d-454ced383b6b","Type":"ContainerStarted","Data":"07f502d670dddd1922839901ced09c783458f415f414e431e3114f58a5dbabdc"} Jul 09 02:00:56.044210 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:56.044178 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:00:56.064029 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:56.063962 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-msjzj" podStartSLOduration=33.708092498 podStartE2EDuration="44.063938853s" totalImagesPullingTime="10.355846355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 02:00:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:00:56.062142033 +0000 UTC m=+85.950304603" watchObservedRunningTime="2026-07-09 02:00:56.063938853 +0000 UTC m=+85.952101419" Jul 09 02:00:58.122748 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.122711 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2lzn6"] Jul 09 02:00:58.126062 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.126039 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.130275 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.130233 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 02:00:58.130518 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.130467 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zz76p\"" Jul 09 02:00:58.131498 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.131474 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 02:00:58.131608 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.131517 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 02:00:58.131821 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.131796 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 02:00:58.155566 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhbb\" (UniqueName: \"kubernetes.io/projected/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-kube-api-access-fbhbb\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.155566 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-metrics-client-ca\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.155827 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-tls\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.155827 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.155827 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-wtmp\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.155827 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-root\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.155827 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-sys\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.156036 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155877 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-accelerators-collector-config\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.156036 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.155902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-textfile\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.256799 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.256760 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-sys\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.256980 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.256823 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-accelerators-collector-config\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.256980 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.256853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-textfile\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.256980 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.256871 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-sys\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.256980 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.256884 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhbb\" (UniqueName: \"kubernetes.io/projected/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-kube-api-access-fbhbb\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.256980 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.256911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-metrics-client-ca\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.256980 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.256955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-tls\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.257298 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:58.257048 2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 02:00:58.257298 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:00:58.257110 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-tls podName:7f77e720-6208-44b4-8eb9-b55ae2d4ce94 nodeName:}" failed. No retries permitted until 2026-07-09 02:00:58.757091087 +0000 UTC m=+88.645253648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-tls") pod "node-exporter-2lzn6" (UID: "7f77e720-6208-44b4-8eb9-b55ae2d4ce94") : secret "node-exporter-tls" not found Jul 09 02:00:58.257298 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.257267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.257479 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.257312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-wtmp\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.257479 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.257308 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-textfile\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.257479 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.257344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-root\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.257479 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.257447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-root\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.257479 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.257458 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-wtmp\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.257479 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.257478 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-accelerators-collector-config\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.257750 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.257590 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-metrics-client-ca\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.259841 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.259821 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.266783 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.266728 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhbb\" (UniqueName: \"kubernetes.io/projected/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-kube-api-access-fbhbb\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.761951 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.761911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-tls\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:58.764686 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:58.764627 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7f77e720-6208-44b4-8eb9-b55ae2d4ce94-node-exporter-tls\") pod \"node-exporter-2lzn6\" (UID: \"7f77e720-6208-44b4-8eb9-b55ae2d4ce94\") " pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:00:59.037336 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:00:59.037189 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2lzn6" Jul 09 02:01:00.951333 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:00.951298 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:01:04.442044 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.441996 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:01:04.518563 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.518520 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:01:04.519138 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.519114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.527118 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.527089 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fz9h6\"" Jul 09 02:01:04.527286 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.527211 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ga07n0h97fpd\"" Jul 09 02:01:04.527533 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.527508 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 02:01:04.527634 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.527514 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 02:01:04.527930 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.527912 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 02:01:04.528157 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.528136 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 02:01:04.528269 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.528136 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 02:01:04.528269 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.528140 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 02:01:04.528896 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.528877 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 02:01:04.530387 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.530357 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 02:01:04.530387 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.530377 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 02:01:04.530704 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.530691 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 02:01:04.530997 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.530980 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 02:01:04.531932 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.531911 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 02:01:04.532027 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.531951 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 02:01:04.533036 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.533015 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 02:01:04.538195 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.538168 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 02:01:04.613707 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.613673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.613906 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.613728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf9pd\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-kube-api-access-xf9pd\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.613906 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.613758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.613906 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.613880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614090 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.613932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614090 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.613999 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614090 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614090 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614367 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614367 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614367 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614367 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-config\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614367 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614367 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614367 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614354 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614717 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614717 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614547 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614717 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614717 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.614717 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.614630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.715894 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.715848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf9pd\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-kube-api-access-xf9pd\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.715894 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.715893 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716158 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716158 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716158 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716158 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-config\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716494 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.716523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716523 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717012 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717012 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717012 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717012 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717012 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716813 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717012 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.716838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717351 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.717082 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717351 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.717307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.717458 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.717428 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.719684 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.719364 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.720462 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.720133 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.720592 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.720566 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.720978 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.720806 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.720978 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.720897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.721672 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.721340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.721672 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.721469 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.722089 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.722035 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.722199 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.722177 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.722401 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.722376 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-web-config\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.722908 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.722878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.723056 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.723016 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.723393 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.723375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-config-out\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.724202 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.724184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-config\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.727779 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.727754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf9pd\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-kube-api-access-xf9pd\") pod \"prometheus-k8s-0\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:04.832478 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:04.832440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:05.241184 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:01:05.241138 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f77e720_6208_44b4_8eb9_b55ae2d4ce94.slice/crio-6d3dcdea4387dfba808fb5e79448e504a7862472acf171c4e51dfb3c01d113ab WatchSource:0}: Error finding container 6d3dcdea4387dfba808fb5e79448e504a7862472acf171c4e51dfb3c01d113ab: Status 404 returned error can't find the container with id 6d3dcdea4387dfba808fb5e79448e504a7862472acf171c4e51dfb3c01d113ab Jul 09 02:01:05.374513 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:01:05.374482 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95768311_0348_408b_a137_66d8a74cf19a.slice/crio-068c8728459c4e15eb18a74e5bc970e2857b5a90fd7fe6f7b596205ac1100b65 WatchSource:0}: Error finding container 068c8728459c4e15eb18a74e5bc970e2857b5a90fd7fe6f7b596205ac1100b65: Status 404 returned error can't find the container with id 068c8728459c4e15eb18a74e5bc970e2857b5a90fd7fe6f7b596205ac1100b65 Jul 09 02:01:05.374513 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:05.374501 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:01:06.083844 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:06.083768 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerStarted","Data":"068c8728459c4e15eb18a74e5bc970e2857b5a90fd7fe6f7b596205ac1100b65"} Jul 09 02:01:06.085828 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:06.085794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-9cmn7" event={"ID":"92d4c717-baf8-4e62-b139-3e67293e89d5","Type":"ContainerStarted","Data":"1f860cffdcf8579667a16ce53275b745cdb6be47bff02708a5caae09af562fff"} Jul 09 02:01:06.086717 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:06.086691 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-9cmn7" Jul 09 02:01:06.088375 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:06.088329 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2lzn6" event={"ID":"7f77e720-6208-44b4-8eb9-b55ae2d4ce94","Type":"ContainerStarted","Data":"6d3dcdea4387dfba808fb5e79448e504a7862472acf171c4e51dfb3c01d113ab"} Jul 09 02:01:06.099793 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:06.099757 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-9cmn7" Jul 09 02:01:06.110616 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:06.110010 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-9cmn7" podStartSLOduration=2.085824719 podStartE2EDuration="22.109989723s" totalImagesPullingTime="20.024165004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:01:06.108966961 +0000 UTC m=+95.997129528" watchObservedRunningTime="2026-07-09 02:01:06.109989723 +0000 UTC m=+95.998152289" Jul 09 02:01:06.314884 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:06.314848 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64b9d58d4f-zkvx7"] Jul 09 02:01:07.095475 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:07.095440 2578 generic.go:356] "Generic (PLEG): container finished" podID="7f77e720-6208-44b4-8eb9-b55ae2d4ce94" containerID="f34c191b81acbe0c05c13fa3dc99937e302e963e2778f931164a987e0fa38fff" exitCode=0 Jul 09 02:01:07.096008 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:07.095534 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2lzn6" event={"ID":"7f77e720-6208-44b4-8eb9-b55ae2d4ce94","Type":"ContainerDied","Data":"f34c191b81acbe0c05c13fa3dc99937e302e963e2778f931164a987e0fa38fff"} Jul 09 02:01:08.103291 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:08.103224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2lzn6" event={"ID":"7f77e720-6208-44b4-8eb9-b55ae2d4ce94","Type":"ContainerStarted","Data":"178ca7ec111477a3e6f071defaba71568cb60944426464a2ad2f6ee63eac6f8d"} Jul 09 02:01:08.103763 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:08.103327 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2lzn6" event={"ID":"7f77e720-6208-44b4-8eb9-b55ae2d4ce94","Type":"ContainerStarted","Data":"5bff25b84e6550b5c7fbb82d6259fb8533c03ecd0dba4fce88c1b55cab1005b3"} Jul 09 02:01:08.105558 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:08.105531 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerStarted","Data":"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f"} Jul 09 02:01:08.128160 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:08.128101 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2lzn6" podStartSLOduration=8.52179042 podStartE2EDuration="10.128084024s" totalImagesPullingTime="1.484054497s" totalInitContainerRuntime="122.239107ms" isStatefulPod=true podCreationTimestamp="2026-07-09 02:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:01:08.125905246 +0000 UTC m=+98.014067846" watchObservedRunningTime="2026-07-09 02:01:08.128084024 +0000 UTC m=+98.016246589" Jul 09 02:01:09.117757 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:09.117716 2578 generic.go:356] "Generic (PLEG): container finished" podID="95768311-0348-408b-a137-66d8a74cf19a" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" exitCode=0 Jul 09 02:01:09.118233 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:09.117801 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerDied","Data":"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f"} Jul 09 02:01:13.997955 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:13.995681 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5488b54ffc-fwq8d"] Jul 09 02:01:14.153059 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:14.153012 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerStarted","Data":"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22"} Jul 09 02:01:14.153268 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:14.153114 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerStarted","Data":"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a"} Jul 09 02:01:14.974153 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:14.974119 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jzpk6" Jul 09 02:01:18.173408 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:18.173375 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerStarted","Data":"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a"} Jul 09 02:01:18.173779 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:18.173439 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerStarted","Data":"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a"} Jul 09 02:01:18.173779 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:18.173474 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerStarted","Data":"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729"} Jul 09 02:01:18.173779 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:18.173497 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerStarted","Data":"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014"} Jul 09 02:01:18.208524 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:18.208468 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.785539611 podStartE2EDuration="14.208449727s" totalImagesPullingTime="10.392377192s" totalInitContainerRuntime="30.532924ms" isStatefulPod=false podCreationTimestamp="2026-07-09 02:01:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:01:18.20662617 +0000 UTC m=+108.094788735" watchObservedRunningTime="2026-07-09 02:01:18.208449727 +0000 UTC m=+108.096612292" Jul 09 02:01:19.832936 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:19.832896 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:01:31.341886 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.341844 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" podUID="6d66a9ae-2d58-4260-a16d-096a4311a43a" containerName="registry" containerID="cri-o://4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866" gracePeriod=30 Jul 09 02:01:31.622514 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.622439 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:01:31.660693 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.660666 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-certificates\") pod \"6d66a9ae-2d58-4260-a16d-096a4311a43a\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " Jul 09 02:01:31.660874 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.660705 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-bound-sa-token\") pod \"6d66a9ae-2d58-4260-a16d-096a4311a43a\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " Jul 09 02:01:31.660874 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.660731 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-trusted-ca\") pod \"6d66a9ae-2d58-4260-a16d-096a4311a43a\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " Jul 09 02:01:31.660874 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.660769 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-installation-pull-secrets\") pod \"6d66a9ae-2d58-4260-a16d-096a4311a43a\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " Jul 09 02:01:31.660874 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.660825 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pczwl\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-kube-api-access-pczwl\") pod \"6d66a9ae-2d58-4260-a16d-096a4311a43a\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " Jul 09 02:01:31.660874 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.660846 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") pod \"6d66a9ae-2d58-4260-a16d-096a4311a43a\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " Jul 09 02:01:31.660874 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.660869 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-image-registry-private-configuration\") pod \"6d66a9ae-2d58-4260-a16d-096a4311a43a\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " Jul 09 02:01:31.661156 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.660916 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d66a9ae-2d58-4260-a16d-096a4311a43a-ca-trust-extracted\") pod \"6d66a9ae-2d58-4260-a16d-096a4311a43a\" (UID: \"6d66a9ae-2d58-4260-a16d-096a4311a43a\") " Jul 09 02:01:31.661622 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.661582 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6d66a9ae-2d58-4260-a16d-096a4311a43a" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:01:31.663503 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.661834 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6d66a9ae-2d58-4260-a16d-096a4311a43a" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:01:31.665029 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.664936 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-kube-api-access-pczwl" (OuterVolumeSpecName: "kube-api-access-pczwl") pod "6d66a9ae-2d58-4260-a16d-096a4311a43a" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a"). InnerVolumeSpecName "kube-api-access-pczwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:01:31.665029 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.664979 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6d66a9ae-2d58-4260-a16d-096a4311a43a" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:01:31.665421 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.665338 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-certificates\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:31.665421 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.665374 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d66a9ae-2d58-4260-a16d-096a4311a43a-trusted-ca\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:31.665850 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.665388 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-installation-pull-secrets\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:31.665850 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.665599 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pczwl\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-kube-api-access-pczwl\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:31.669711 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.669678 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6d66a9ae-2d58-4260-a16d-096a4311a43a" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:01:31.670923 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.670815 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6d66a9ae-2d58-4260-a16d-096a4311a43a" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:01:31.671262 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.671208 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6d66a9ae-2d58-4260-a16d-096a4311a43a" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:01:31.676336 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.676309 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d66a9ae-2d58-4260-a16d-096a4311a43a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6d66a9ae-2d58-4260-a16d-096a4311a43a" (UID: "6d66a9ae-2d58-4260-a16d-096a4311a43a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:01:31.766559 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.766517 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-bound-sa-token\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:31.766559 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.766549 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d66a9ae-2d58-4260-a16d-096a4311a43a-registry-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:31.766559 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.766562 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d66a9ae-2d58-4260-a16d-096a4311a43a-image-registry-private-configuration\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:31.766796 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:31.766575 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d66a9ae-2d58-4260-a16d-096a4311a43a-ca-trust-extracted\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:32.231842 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.231800 2578 generic.go:356] "Generic (PLEG): container finished" podID="6d66a9ae-2d58-4260-a16d-096a4311a43a" containerID="4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866" exitCode=0 Jul 09 02:01:32.232017 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.231867 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" Jul 09 02:01:32.232017 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.231870 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" event={"ID":"6d66a9ae-2d58-4260-a16d-096a4311a43a","Type":"ContainerDied","Data":"4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866"} Jul 09 02:01:32.232017 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.231945 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b9d58d4f-zkvx7" event={"ID":"6d66a9ae-2d58-4260-a16d-096a4311a43a","Type":"ContainerDied","Data":"8f640153884aeea7a95f430c1c0174f193a22817e8f71ccbae41c2387be5d49f"} Jul 09 02:01:32.232017 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.231972 2578 scope.go:117] "RemoveContainer" containerID="4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866" Jul 09 02:01:32.241028 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.241005 2578 scope.go:117] "RemoveContainer" containerID="4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866" Jul 09 02:01:32.241424 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:01:32.241394 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866\": container with ID starting with 4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866 not found: ID does not exist" containerID="4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866" Jul 09 02:01:32.241487 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.241433 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866"} err="failed to get container status \"4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866\": rpc error: code = NotFound desc = could not find container \"4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866\": container with ID starting with 4f81ad28b5542c7fde30f7232a93478f627a134d0040487bd0c36a85e55a7866 not found: ID does not exist" Jul 09 02:01:32.255054 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.255026 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64b9d58d4f-zkvx7"] Jul 09 02:01:32.259863 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.259828 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64b9d58d4f-zkvx7"] Jul 09 02:01:32.579886 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:32.579798 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d66a9ae-2d58-4260-a16d-096a4311a43a" path="/var/lib/kubelet/pods/6d66a9ae-2d58-4260-a16d-096a4311a43a/volumes" Jul 09 02:01:39.029077 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.029009 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5488b54ffc-fwq8d" podUID="54465bc4-c858-45a7-820a-656af3984054" containerName="console" containerID="cri-o://f2f3a7a63b213d5b2a31c2915bfe6d5f326d49f0408662db8a4f59aeae04286a" gracePeriod=15 Jul 09 02:01:39.259988 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.259952 2578 generic.go:356] "Generic (PLEG): container finished" podID="54465bc4-c858-45a7-820a-656af3984054" containerID="f2f3a7a63b213d5b2a31c2915bfe6d5f326d49f0408662db8a4f59aeae04286a" exitCode=0 Jul 09 02:01:39.260161 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.260030 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5488b54ffc-fwq8d" event={"ID":"54465bc4-c858-45a7-820a-656af3984054","Type":"ContainerDied","Data":"f2f3a7a63b213d5b2a31c2915bfe6d5f326d49f0408662db8a4f59aeae04286a"} Jul 09 02:01:39.301729 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.301647 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:01:39.330903 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.330871 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-console-config\") pod \"54465bc4-c858-45a7-820a-656af3984054\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " Jul 09 02:01:39.330903 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.330912 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xs47\" (UniqueName: \"kubernetes.io/projected/54465bc4-c858-45a7-820a-656af3984054-kube-api-access-8xs47\") pod \"54465bc4-c858-45a7-820a-656af3984054\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " Jul 09 02:01:39.331152 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.330971 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-service-ca\") pod \"54465bc4-c858-45a7-820a-656af3984054\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " Jul 09 02:01:39.331152 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.330995 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-oauth-serving-cert\") pod \"54465bc4-c858-45a7-820a-656af3984054\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " Jul 09 02:01:39.331152 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.331020 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-serving-cert\") pod \"54465bc4-c858-45a7-820a-656af3984054\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " Jul 09 02:01:39.331152 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.331062 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-oauth-config\") pod \"54465bc4-c858-45a7-820a-656af3984054\" (UID: \"54465bc4-c858-45a7-820a-656af3984054\") " Jul 09 02:01:39.331394 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.331370 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-console-config" (OuterVolumeSpecName: "console-config") pod "54465bc4-c858-45a7-820a-656af3984054" (UID: "54465bc4-c858-45a7-820a-656af3984054"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:01:39.331701 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.331657 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "54465bc4-c858-45a7-820a-656af3984054" (UID: "54465bc4-c858-45a7-820a-656af3984054"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:01:39.331811 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.331675 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-service-ca" (OuterVolumeSpecName: "service-ca") pod "54465bc4-c858-45a7-820a-656af3984054" (UID: "54465bc4-c858-45a7-820a-656af3984054"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:01:39.334154 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.334126 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54465bc4-c858-45a7-820a-656af3984054-kube-api-access-8xs47" (OuterVolumeSpecName: "kube-api-access-8xs47") pod "54465bc4-c858-45a7-820a-656af3984054" (UID: "54465bc4-c858-45a7-820a-656af3984054"). InnerVolumeSpecName "kube-api-access-8xs47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:01:39.334288 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.334196 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "54465bc4-c858-45a7-820a-656af3984054" (UID: "54465bc4-c858-45a7-820a-656af3984054"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:01:39.334503 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.334479 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "54465bc4-c858-45a7-820a-656af3984054" (UID: "54465bc4-c858-45a7-820a-656af3984054"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:01:39.432354 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.432313 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-oauth-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:39.432354 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.432349 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-console-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:39.432354 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.432360 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xs47\" (UniqueName: \"kubernetes.io/projected/54465bc4-c858-45a7-820a-656af3984054-kube-api-access-8xs47\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:39.432354 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.432368 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-service-ca\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:39.432629 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.432377 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/54465bc4-c858-45a7-820a-656af3984054-oauth-serving-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:39.432629 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:39.432385 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/54465bc4-c858-45a7-820a-656af3984054-console-serving-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:01:40.265144 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:40.265108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5488b54ffc-fwq8d" event={"ID":"54465bc4-c858-45a7-820a-656af3984054","Type":"ContainerDied","Data":"f057f87801fbe8aae714bbbea296db896569d970f06cc4fce635688880eb6127"} Jul 09 02:01:40.265144 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:40.265131 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5488b54ffc-fwq8d" Jul 09 02:01:40.265652 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:40.265167 2578 scope.go:117] "RemoveContainer" containerID="f2f3a7a63b213d5b2a31c2915bfe6d5f326d49f0408662db8a4f59aeae04286a" Jul 09 02:01:40.288911 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:40.288879 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5488b54ffc-fwq8d"] Jul 09 02:01:40.300952 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:40.300926 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5488b54ffc-fwq8d"] Jul 09 02:01:40.579068 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:40.578988 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54465bc4-c858-45a7-820a-656af3984054" path="/var/lib/kubelet/pods/54465bc4-c858-45a7-820a-656af3984054/volumes" Jul 09 02:01:42.275210 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.275174 2578 generic.go:356] "Generic (PLEG): container finished" podID="8290af00-40ea-4657-bf4e-92137e8bc739" containerID="dae972878f351922c4f95cb8e36dc2677e6e2171a43dcae085b6ca59ff143c8d" exitCode=0 Jul 09 02:01:42.275674 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.275264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" event={"ID":"8290af00-40ea-4657-bf4e-92137e8bc739","Type":"ContainerDied","Data":"dae972878f351922c4f95cb8e36dc2677e6e2171a43dcae085b6ca59ff143c8d"} Jul 09 02:01:42.275674 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.275656 2578 scope.go:117] "RemoveContainer" containerID="dae972878f351922c4f95cb8e36dc2677e6e2171a43dcae085b6ca59ff143c8d" Jul 09 02:01:42.276750 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.276730 2578 generic.go:356] "Generic (PLEG): container finished" podID="71731d5b-3313-4046-8579-f828a13ba12c" containerID="1d4230af9a077acae1bb0413eec711e0b09f50437f0c3bdfca54ee0cd44fb456" exitCode=0 Jul 09 02:01:42.276848 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.276798 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" event={"ID":"71731d5b-3313-4046-8579-f828a13ba12c","Type":"ContainerDied","Data":"1d4230af9a077acae1bb0413eec711e0b09f50437f0c3bdfca54ee0cd44fb456"} Jul 09 02:01:42.277129 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.277115 2578 scope.go:117] "RemoveContainer" containerID="1d4230af9a077acae1bb0413eec711e0b09f50437f0c3bdfca54ee0cd44fb456" Jul 09 02:01:42.278233 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.278216 2578 generic.go:356] "Generic (PLEG): container finished" podID="13021762-a6ad-46ab-a4de-dc1838a6662e" containerID="7770b16811a1522a1710a77f23ec0da1f677f4b84f8e8f6e3c171067a63ed9e1" exitCode=0 Jul 09 02:01:42.278346 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.278289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" event={"ID":"13021762-a6ad-46ab-a4de-dc1838a6662e","Type":"ContainerDied","Data":"7770b16811a1522a1710a77f23ec0da1f677f4b84f8e8f6e3c171067a63ed9e1"} Jul 09 02:01:42.278563 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:42.278547 2578 scope.go:117] "RemoveContainer" containerID="7770b16811a1522a1710a77f23ec0da1f677f4b84f8e8f6e3c171067a63ed9e1" Jul 09 02:01:43.284965 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:43.284928 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-n8hg2" event={"ID":"13021762-a6ad-46ab-a4de-dc1838a6662e","Type":"ContainerStarted","Data":"1dfe4922b51447e52762df1213e9cc12110e5753a49ac844fb15a4439833d7d7"} Jul 09 02:01:43.286900 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:43.286870 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-4x69v" event={"ID":"8290af00-40ea-4657-bf4e-92137e8bc739","Type":"ContainerStarted","Data":"6bb845837b8edb41be2114c63b7fc27a64ab52780d76157c53cff2aa32ab2835"} Jul 09 02:01:43.288662 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:01:43.288638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-htm8p" event={"ID":"71731d5b-3313-4046-8579-f828a13ba12c","Type":"ContainerStarted","Data":"360035360d14ef64aa94f2763619c6d82f4bcde071e8f3eb4ab414e4868f45cf"} Jul 09 02:02:04.833101 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:04.833067 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:04.848649 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:04.848622 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:05.384101 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:05.384071 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:22.930395 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:22.930317 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:02:22.930829 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:22.930627 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="prometheus" containerID="cri-o://ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" gracePeriod=600 Jul 09 02:02:22.930829 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:22.930656 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy" containerID="cri-o://8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" gracePeriod=600 Jul 09 02:02:22.930829 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:22.930656 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="thanos-sidecar" containerID="cri-o://0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" gracePeriod=600 Jul 09 02:02:22.930829 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:22.930680 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy-web" containerID="cri-o://46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" gracePeriod=600 Jul 09 02:02:22.930829 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:22.930688 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="config-reloader" containerID="cri-o://ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" gracePeriod=600 Jul 09 02:02:22.930829 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:22.930658 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" gracePeriod=600 Jul 09 02:02:23.176180 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.176154 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.214278 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214229 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-tls-assets\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214278 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214270 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-config-out\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214289 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-trusted-ca-bundle\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214308 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-metrics-client-ca\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214330 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-0\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214358 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-1\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214384 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-kubelet-serving-ca-bundle\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214418 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-grpc-tls\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214444 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf9pd\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-kube-api-access-xf9pd\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214469 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214512 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-config\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214996 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214546 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-tls\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214996 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214573 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-web-config\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.214996 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.214600 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-thanos-prometheus-http-client-file\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.215162 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.215108 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:02:23.215457 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.215418 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:02:23.215457 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.215437 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.215962 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216018 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-metrics-client-certs\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216064 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-kube-rbac-proxy\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216104 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216129 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-2\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216159 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-serving-certs-ca-bundle\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216185 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-db\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216209 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"95768311-0348-408b-a137-66d8a74cf19a\" (UID: \"95768311-0348-408b-a137-66d8a74cf19a\") " Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216468 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216489 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-metrics-client-ca\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216503 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216517 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.217071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216846 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:02:23.217810 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.217102 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-config-out" (OuterVolumeSpecName: "config-out") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:02:23.217810 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.217168 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:02:23.217810 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.217537 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:02:23.218310 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.216533 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.219291 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.219232 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.219597 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.219374 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:02:23.220092 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.219983 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.220193 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.220116 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.220359 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.220327 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-kube-api-access-xf9pd" (OuterVolumeSpecName: "kube-api-access-xf9pd") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "kube-api-access-xf9pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:02:23.220649 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.220623 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.220732 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.220670 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.221697 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.221210 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.221697 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.221238 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.221854 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.221826 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-config" (OuterVolumeSpecName: "config") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.232153 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.232126 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-web-config" (OuterVolumeSpecName: "web-config") pod "95768311-0348-408b-a137-66d8a74cf19a" (UID: "95768311-0348-408b-a137-66d8a74cf19a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:02:23.318714 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318675 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318714 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318707 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95768311-0348-408b-a137-66d8a74cf19a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318714 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318718 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-prometheus-k8s-db\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318727 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318737 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-tls-assets\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318745 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95768311-0348-408b-a137-66d8a74cf19a-config-out\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318752 2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-grpc-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318760 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf9pd\" (UniqueName: \"kubernetes.io/projected/95768311-0348-408b-a137-66d8a74cf19a-kube-api-access-xf9pd\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318768 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318776 2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318784 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318792 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-web-config\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318800 2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318808 2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-metrics-client-certs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.318942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.318817 2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95768311-0348-408b-a137-66d8a74cf19a-secret-kube-rbac-proxy\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:02:23.440761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440719 2578 generic.go:356] "Generic (PLEG): container finished" podID="95768311-0348-408b-a137-66d8a74cf19a" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" exitCode=0 Jul 09 02:02:23.440761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440748 2578 generic.go:356] "Generic (PLEG): container finished" podID="95768311-0348-408b-a137-66d8a74cf19a" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" exitCode=0 Jul 09 02:02:23.440761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440756 2578 generic.go:356] "Generic (PLEG): container finished" podID="95768311-0348-408b-a137-66d8a74cf19a" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" exitCode=0 Jul 09 02:02:23.440761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440761 2578 generic.go:356] "Generic (PLEG): container finished" podID="95768311-0348-408b-a137-66d8a74cf19a" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" exitCode=0 Jul 09 02:02:23.440761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440767 2578 generic.go:356] "Generic (PLEG): container finished" podID="95768311-0348-408b-a137-66d8a74cf19a" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" exitCode=0 Jul 09 02:02:23.440761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440773 2578 generic.go:356] "Generic (PLEG): container finished" podID="95768311-0348-408b-a137-66d8a74cf19a" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" exitCode=0 Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440801 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerDied","Data":"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a"} Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerDied","Data":"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a"} Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440967 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerDied","Data":"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729"} Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.440991 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerDied","Data":"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014"} Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.441007 2578 scope.go:117] "RemoveContainer" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.441014 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerDied","Data":"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22"} Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.441040 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerDied","Data":"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a"} Jul 09 02:02:23.441076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.441063 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95768311-0348-408b-a137-66d8a74cf19a","Type":"ContainerDied","Data":"068c8728459c4e15eb18a74e5bc970e2857b5a90fd7fe6f7b596205ac1100b65"} Jul 09 02:02:23.449551 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.449534 2578 scope.go:117] "RemoveContainer" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" Jul 09 02:02:23.456618 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.456601 2578 scope.go:117] "RemoveContainer" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" Jul 09 02:02:23.463275 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.463238 2578 scope.go:117] "RemoveContainer" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" Jul 09 02:02:23.467603 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.467550 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:02:23.470844 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.470818 2578 scope.go:117] "RemoveContainer" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" Jul 09 02:02:23.472839 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.472821 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:02:23.478093 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.478073 2578 scope.go:117] "RemoveContainer" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" Jul 09 02:02:23.485535 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.485514 2578 scope.go:117] "RemoveContainer" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" Jul 09 02:02:23.492470 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.492449 2578 scope.go:117] "RemoveContainer" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" Jul 09 02:02:23.492746 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:02:23.492727 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": container with ID starting with b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a not found: ID does not exist" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" Jul 09 02:02:23.492798 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.492756 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a"} err="failed to get container status \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": rpc error: code = NotFound desc = could not find container \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": container with ID starting with b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a not found: ID does not exist" Jul 09 02:02:23.492798 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.492776 2578 scope.go:117] "RemoveContainer" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" Jul 09 02:02:23.493063 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:02:23.493044 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": container with ID starting with 8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a not found: ID does not exist" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" Jul 09 02:02:23.493135 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.493077 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a"} err="failed to get container status \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": rpc error: code = NotFound desc = could not find container \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": container with ID starting with 8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a not found: ID does not exist" Jul 09 02:02:23.493135 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.493100 2578 scope.go:117] "RemoveContainer" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" Jul 09 02:02:23.493366 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:02:23.493349 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": container with ID starting with 46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729 not found: ID does not exist" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" Jul 09 02:02:23.493410 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.493371 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729"} err="failed to get container status \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": rpc error: code = NotFound desc = could not find container \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": container with ID starting with 46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729 not found: ID does not exist" Jul 09 02:02:23.493410 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.493386 2578 scope.go:117] "RemoveContainer" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" Jul 09 02:02:23.493602 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:02:23.493585 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": container with ID starting with 0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014 not found: ID does not exist" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" Jul 09 02:02:23.493641 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.493608 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014"} err="failed to get container status \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": rpc error: code = NotFound desc = could not find container \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": container with ID starting with 0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014 not found: ID does not exist" Jul 09 02:02:23.493641 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.493624 2578 scope.go:117] "RemoveContainer" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" Jul 09 02:02:23.493844 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:02:23.493829 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": container with ID starting with ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22 not found: ID does not exist" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" Jul 09 02:02:23.493887 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.493847 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22"} err="failed to get container status \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": rpc error: code = NotFound desc = could not find container \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": container with ID starting with ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22 not found: ID does not exist" Jul 09 02:02:23.493887 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.493859 2578 scope.go:117] "RemoveContainer" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" Jul 09 02:02:23.494053 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:02:23.494040 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": container with ID starting with ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a not found: ID does not exist" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" Jul 09 02:02:23.494086 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494055 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a"} err="failed to get container status \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": rpc error: code = NotFound desc = could not find container \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": container with ID starting with ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a not found: ID does not exist" Jul 09 02:02:23.494086 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494066 2578 scope.go:117] "RemoveContainer" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" Jul 09 02:02:23.494288 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:02:23.494266 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": container with ID starting with ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f not found: ID does not exist" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" Jul 09 02:02:23.494363 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494295 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f"} err="failed to get container status \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": rpc error: code = NotFound desc = could not find container \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": container with ID starting with ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f not found: ID does not exist" Jul 09 02:02:23.494363 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494316 2578 scope.go:117] "RemoveContainer" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" Jul 09 02:02:23.494577 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494558 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a"} err="failed to get container status \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": rpc error: code = NotFound desc = could not find container \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": container with ID starting with b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a not found: ID does not exist" Jul 09 02:02:23.494577 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494576 2578 scope.go:117] "RemoveContainer" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" Jul 09 02:02:23.494783 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494766 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a"} err="failed to get container status \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": rpc error: code = NotFound desc = could not find container \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": container with ID starting with 8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a not found: ID does not exist" Jul 09 02:02:23.494821 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494783 2578 scope.go:117] "RemoveContainer" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" Jul 09 02:02:23.494960 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494945 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729"} err="failed to get container status \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": rpc error: code = NotFound desc = could not find container \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": container with ID starting with 46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729 not found: ID does not exist" Jul 09 02:02:23.495000 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.494960 2578 scope.go:117] "RemoveContainer" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" Jul 09 02:02:23.495134 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495116 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014"} err="failed to get container status \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": rpc error: code = NotFound desc = could not find container \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": container with ID starting with 0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014 not found: ID does not exist" Jul 09 02:02:23.495174 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495134 2578 scope.go:117] "RemoveContainer" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" Jul 09 02:02:23.495411 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495394 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22"} err="failed to get container status \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": rpc error: code = NotFound desc = could not find container \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": container with ID starting with ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22 not found: ID does not exist" Jul 09 02:02:23.495411 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495411 2578 scope.go:117] "RemoveContainer" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" Jul 09 02:02:23.495601 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495583 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a"} err="failed to get container status \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": rpc error: code = NotFound desc = could not find container \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": container with ID starting with ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a not found: ID does not exist" Jul 09 02:02:23.495641 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495603 2578 scope.go:117] "RemoveContainer" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" Jul 09 02:02:23.495787 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495765 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f"} err="failed to get container status \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": rpc error: code = NotFound desc = could not find container \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": container with ID starting with ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f not found: ID does not exist" Jul 09 02:02:23.495860 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495788 2578 scope.go:117] "RemoveContainer" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" Jul 09 02:02:23.495986 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495961 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a"} err="failed to get container status \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": rpc error: code = NotFound desc = could not find container \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": container with ID starting with b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a not found: ID does not exist" Jul 09 02:02:23.496027 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.495987 2578 scope.go:117] "RemoveContainer" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" Jul 09 02:02:23.496162 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496148 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a"} err="failed to get container status \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": rpc error: code = NotFound desc = could not find container \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": container with ID starting with 8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a not found: ID does not exist" Jul 09 02:02:23.496198 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496163 2578 scope.go:117] "RemoveContainer" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" Jul 09 02:02:23.496351 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496336 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729"} err="failed to get container status \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": rpc error: code = NotFound desc = could not find container \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": container with ID starting with 46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729 not found: ID does not exist" Jul 09 02:02:23.496392 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496351 2578 scope.go:117] "RemoveContainer" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" Jul 09 02:02:23.496539 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496524 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014"} err="failed to get container status \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": rpc error: code = NotFound desc = could not find container \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": container with ID starting with 0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014 not found: ID does not exist" Jul 09 02:02:23.496583 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496540 2578 scope.go:117] "RemoveContainer" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" Jul 09 02:02:23.496714 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496698 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22"} err="failed to get container status \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": rpc error: code = NotFound desc = could not find container \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": container with ID starting with ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22 not found: ID does not exist" Jul 09 02:02:23.496751 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496715 2578 scope.go:117] "RemoveContainer" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" Jul 09 02:02:23.496879 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496859 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a"} err="failed to get container status \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": rpc error: code = NotFound desc = could not find container \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": container with ID starting with ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a not found: ID does not exist" Jul 09 02:02:23.496879 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.496879 2578 scope.go:117] "RemoveContainer" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" Jul 09 02:02:23.497097 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497081 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f"} err="failed to get container status \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": rpc error: code = NotFound desc = could not find container \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": container with ID starting with ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f not found: ID does not exist" Jul 09 02:02:23.497097 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497096 2578 scope.go:117] "RemoveContainer" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" Jul 09 02:02:23.497289 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497274 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a"} err="failed to get container status \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": rpc error: code = NotFound desc = could not find container \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": container with ID starting with b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a not found: ID does not exist" Jul 09 02:02:23.497336 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497290 2578 scope.go:117] "RemoveContainer" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" Jul 09 02:02:23.497467 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497452 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a"} err="failed to get container status \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": rpc error: code = NotFound desc = could not find container \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": container with ID starting with 8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a not found: ID does not exist" Jul 09 02:02:23.497467 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497467 2578 scope.go:117] "RemoveContainer" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" Jul 09 02:02:23.497639 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497624 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729"} err="failed to get container status \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": rpc error: code = NotFound desc = could not find container \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": container with ID starting with 46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729 not found: ID does not exist" Jul 09 02:02:23.497679 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497640 2578 scope.go:117] "RemoveContainer" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" Jul 09 02:02:23.497823 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497807 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014"} err="failed to get container status \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": rpc error: code = NotFound desc = could not find container \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": container with ID starting with 0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014 not found: ID does not exist" Jul 09 02:02:23.497864 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497823 2578 scope.go:117] "RemoveContainer" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" Jul 09 02:02:23.497986 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497973 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22"} err="failed to get container status \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": rpc error: code = NotFound desc = could not find container \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": container with ID starting with ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22 not found: ID does not exist" Jul 09 02:02:23.498028 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.497988 2578 scope.go:117] "RemoveContainer" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" Jul 09 02:02:23.498148 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498134 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a"} err="failed to get container status \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": rpc error: code = NotFound desc = could not find container \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": container with ID starting with ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a not found: ID does not exist" Jul 09 02:02:23.498189 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498149 2578 scope.go:117] "RemoveContainer" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" Jul 09 02:02:23.498345 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498324 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f"} err="failed to get container status \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": rpc error: code = NotFound desc = could not find container \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": container with ID starting with ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f not found: ID does not exist" Jul 09 02:02:23.498345 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498343 2578 scope.go:117] "RemoveContainer" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" Jul 09 02:02:23.498541 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498527 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a"} err="failed to get container status \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": rpc error: code = NotFound desc = could not find container \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": container with ID starting with b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a not found: ID does not exist" Jul 09 02:02:23.498590 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498541 2578 scope.go:117] "RemoveContainer" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" Jul 09 02:02:23.498774 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498751 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a"} err="failed to get container status \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": rpc error: code = NotFound desc = could not find container \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": container with ID starting with 8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a not found: ID does not exist" Jul 09 02:02:23.498838 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498774 2578 scope.go:117] "RemoveContainer" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" Jul 09 02:02:23.498972 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498955 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729"} err="failed to get container status \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": rpc error: code = NotFound desc = could not find container \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": container with ID starting with 46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729 not found: ID does not exist" Jul 09 02:02:23.499006 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.498974 2578 scope.go:117] "RemoveContainer" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" Jul 09 02:02:23.499158 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499139 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014"} err="failed to get container status \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": rpc error: code = NotFound desc = could not find container \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": container with ID starting with 0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014 not found: ID does not exist" Jul 09 02:02:23.499227 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499159 2578 scope.go:117] "RemoveContainer" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" Jul 09 02:02:23.499422 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499406 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22"} err="failed to get container status \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": rpc error: code = NotFound desc = could not find container \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": container with ID starting with ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22 not found: ID does not exist" Jul 09 02:02:23.499480 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499422 2578 scope.go:117] "RemoveContainer" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" Jul 09 02:02:23.499641 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499625 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a"} err="failed to get container status \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": rpc error: code = NotFound desc = could not find container \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": container with ID starting with ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a not found: ID does not exist" Jul 09 02:02:23.499685 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499641 2578 scope.go:117] "RemoveContainer" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" Jul 09 02:02:23.499820 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499803 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f"} err="failed to get container status \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": rpc error: code = NotFound desc = could not find container \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": container with ID starting with ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f not found: ID does not exist" Jul 09 02:02:23.499869 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499820 2578 scope.go:117] "RemoveContainer" containerID="b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a" Jul 09 02:02:23.500001 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.499984 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a"} err="failed to get container status \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": rpc error: code = NotFound desc = could not find container \"b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a\": container with ID starting with b18b83c79100c48e71cef6e890dc627bb9f5f465c7dca3bddc16f6967c382d9a not found: ID does not exist" Jul 09 02:02:23.500041 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500002 2578 scope.go:117] "RemoveContainer" containerID="8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a" Jul 09 02:02:23.500177 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500161 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a"} err="failed to get container status \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": rpc error: code = NotFound desc = could not find container \"8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a\": container with ID starting with 8046cbab608cbfa93be10cfb504e4ec6049abedfb5ef7fffbfbde6c64b98fe4a not found: ID does not exist" Jul 09 02:02:23.500215 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500176 2578 scope.go:117] "RemoveContainer" containerID="46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729" Jul 09 02:02:23.500355 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500341 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729"} err="failed to get container status \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": rpc error: code = NotFound desc = could not find container \"46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729\": container with ID starting with 46bd9b14ff24e60c78b4284935a7317933a690456f2543d1d870d3c566010729 not found: ID does not exist" Jul 09 02:02:23.500397 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500355 2578 scope.go:117] "RemoveContainer" containerID="0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014" Jul 09 02:02:23.500522 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500507 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014"} err="failed to get container status \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": rpc error: code = NotFound desc = could not find container \"0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014\": container with ID starting with 0ec602cede72f0bf0dcec843b6f03cb03d348f09adf763d0fe34fbfa8cd1d014 not found: ID does not exist" Jul 09 02:02:23.500522 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500522 2578 scope.go:117] "RemoveContainer" containerID="ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22" Jul 09 02:02:23.500703 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500689 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22"} err="failed to get container status \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": rpc error: code = NotFound desc = could not find container \"ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22\": container with ID starting with ec7dfc7a8b493bb567644be19611d8da2f494855f9abc19586c3df436d338e22 not found: ID does not exist" Jul 09 02:02:23.500747 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500703 2578 scope.go:117] "RemoveContainer" containerID="ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a" Jul 09 02:02:23.500925 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500905 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a"} err="failed to get container status \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": rpc error: code = NotFound desc = could not find container \"ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a\": container with ID starting with ff19dc077017fa2b995c8b5665d287740fe05304740816fa58a662ebe83be54a not found: ID does not exist" Jul 09 02:02:23.500995 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.500926 2578 scope.go:117] "RemoveContainer" containerID="ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f" Jul 09 02:02:23.501219 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.501200 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f"} err="failed to get container status \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": rpc error: code = NotFound desc = could not find container \"ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f\": container with ID starting with ee0a54cf1383482b0427a77a4583c868f64548e3a750b67f168269185290a53f not found: ID does not exist" Jul 09 02:02:23.503348 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503330 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:02:23.503612 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503599 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54465bc4-c858-45a7-820a-656af3984054" containerName="console" Jul 09 02:02:23.503652 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503614 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="54465bc4-c858-45a7-820a-656af3984054" containerName="console" Jul 09 02:02:23.503652 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503628 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="thanos-sidecar" Jul 09 02:02:23.503652 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503633 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="thanos-sidecar" Jul 09 02:02:23.503652 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503644 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="init-config-reloader" Jul 09 02:02:23.503652 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503651 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="init-config-reloader" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503659 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="prometheus" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503664 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="prometheus" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503669 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="config-reloader" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503674 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="config-reloader" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503680 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy-thanos" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503685 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy-thanos" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503692 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d66a9ae-2d58-4260-a16d-096a4311a43a" containerName="registry" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503697 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d66a9ae-2d58-4260-a16d-096a4311a43a" containerName="registry" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503703 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy-web" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503708 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy-web" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503713 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503718 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503768 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="54465bc4-c858-45a7-820a-656af3984054" containerName="console" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503775 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy-web" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503780 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d66a9ae-2d58-4260-a16d-096a4311a43a" containerName="registry" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503784 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="thanos-sidecar" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503789 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="prometheus" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503794 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy" Jul 09 02:02:23.503791 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503798 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="kube-rbac-proxy-thanos" Jul 09 02:02:23.504319 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.503804 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="95768311-0348-408b-a137-66d8a74cf19a" containerName="config-reloader" Jul 09 02:02:23.507467 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.507451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.510752 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.510733 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 02:02:23.510879 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.510862 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 02:02:23.510953 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.510874 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 02:02:23.511003 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.510991 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 02:02:23.511053 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511022 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ga07n0h97fpd\"" Jul 09 02:02:23.511309 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511292 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fz9h6\"" Jul 09 02:02:23.511378 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511311 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 02:02:23.511571 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511556 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 02:02:23.511636 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511574 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 02:02:23.511636 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511557 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 02:02:23.511733 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511696 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 02:02:23.511891 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511876 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 02:02:23.511942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.511878 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 02:02:23.512207 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.512193 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 02:02:23.512810 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.512796 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 02:02:23.515876 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.515859 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 02:02:23.518134 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.518117 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 02:02:23.526772 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.526751 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:02:23.622342 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622342 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g27vk\" (UniqueName: \"kubernetes.io/projected/7ee28e79-757c-438f-a450-afb20a69e324-kube-api-access-g27vk\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622536 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622536 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622536 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-config\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622536 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622536 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee28e79-757c-438f-a450-afb20a69e324-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622664 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622851 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622851 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622851 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee28e79-757c-438f-a450-afb20a69e324-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622851 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622790 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622851 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.622851 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622823 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.623021 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.623021 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.622872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723605 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-config\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723605 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723605 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723605 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee28e79-757c-438f-a450-afb20a69e324-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee28e79-757c-438f-a450-afb20a69e324-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.723899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723946 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.723998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.724024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.724062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.724068 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.724083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g27vk\" (UniqueName: \"kubernetes.io/projected/7ee28e79-757c-438f-a450-afb20a69e324-kube-api-access-g27vk\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.724157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.724193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.724220 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.724958 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.724903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.725479 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.725450 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.726757 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.726727 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee28e79-757c-438f-a450-afb20a69e324-config-out\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.726955 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.726922 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-config\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.727025 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.726966 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.727025 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.726988 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-web-config\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.727025 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.726998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.728021 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.727696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.728021 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.727753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.728021 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.727968 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.728673 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.728295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.728766 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.728746 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.729401 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.729375 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.730229 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.730199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.730332 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.730290 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee28e79-757c-438f-a450-afb20a69e324-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.733052 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.731081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7ee28e79-757c-438f-a450-afb20a69e324-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.733052 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.732208 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7ee28e79-757c-438f-a450-afb20a69e324-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.734928 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.734908 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g27vk\" (UniqueName: \"kubernetes.io/projected/7ee28e79-757c-438f-a450-afb20a69e324-kube-api-access-g27vk\") pod \"prometheus-k8s-0\" (UID: \"7ee28e79-757c-438f-a450-afb20a69e324\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.817847 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.817802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:02:23.961769 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:23.961685 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 02:02:23.964148 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:02:23.964120 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee28e79_757c_438f_a450_afb20a69e324.slice/crio-e7a5383847e1dc2f5664703923e265624f0a62951edd1528b118bc2f56ac4192 WatchSource:0}: Error finding container e7a5383847e1dc2f5664703923e265624f0a62951edd1528b118bc2f56ac4192: Status 404 returned error can't find the container with id e7a5383847e1dc2f5664703923e265624f0a62951edd1528b118bc2f56ac4192 Jul 09 02:02:24.447478 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:24.447384 2578 generic.go:356] "Generic (PLEG): container finished" podID="7ee28e79-757c-438f-a450-afb20a69e324" containerID="66f44f9dcd808a1cd36a5ae2c8be8fe608e7b1f9df28f7a2a74f8eec150c79dc" exitCode=0 Jul 09 02:02:24.447478 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:24.447438 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ee28e79-757c-438f-a450-afb20a69e324","Type":"ContainerDied","Data":"66f44f9dcd808a1cd36a5ae2c8be8fe608e7b1f9df28f7a2a74f8eec150c79dc"} Jul 09 02:02:24.447648 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:24.447503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ee28e79-757c-438f-a450-afb20a69e324","Type":"ContainerStarted","Data":"e7a5383847e1dc2f5664703923e265624f0a62951edd1528b118bc2f56ac4192"} Jul 09 02:02:24.582499 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:24.582461 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95768311-0348-408b-a137-66d8a74cf19a" path="/var/lib/kubelet/pods/95768311-0348-408b-a137-66d8a74cf19a/volumes" Jul 09 02:02:25.454773 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:25.454741 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ee28e79-757c-438f-a450-afb20a69e324","Type":"ContainerStarted","Data":"d7c080e063f81370384eb62cecf8893e9f3739fc060c30bc2f39acadba386047"} Jul 09 02:02:25.455150 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:25.454805 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ee28e79-757c-438f-a450-afb20a69e324","Type":"ContainerStarted","Data":"d95d28e91524e990dccb21e144507ec68b889d61ad91efd8671a9e55f9175386"} Jul 09 02:02:25.455150 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:25.454837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ee28e79-757c-438f-a450-afb20a69e324","Type":"ContainerStarted","Data":"96af7fd2b92da66c9246e47e5b9b9c7109d73378f12c06f261433b02415addfe"} Jul 09 02:02:25.455150 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:25.454860 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ee28e79-757c-438f-a450-afb20a69e324","Type":"ContainerStarted","Data":"36b598395f062f62dacc934a44bfe47fd93da1c9be729620e3835c0607172181"} Jul 09 02:02:25.455150 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:25.454883 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ee28e79-757c-438f-a450-afb20a69e324","Type":"ContainerStarted","Data":"6c4141031657b0f3d7e7a8ce489bfa3cbda3b6361b361636fb7b20978e65f1e9"} Jul 09 02:02:25.455150 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:25.454908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7ee28e79-757c-438f-a450-afb20a69e324","Type":"ContainerStarted","Data":"a609155fc19376a339c04b2086316eddd4355803eef300faf0a675fba9c7999a"} Jul 09 02:02:25.487029 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:25.486970 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.466599258 podStartE2EDuration="2.486948735s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.349477ms" isStatefulPod=false podCreationTimestamp="2026-07-09 02:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:02:25.485290839 +0000 UTC m=+175.373453406" watchObservedRunningTime="2026-07-09 02:02:25.486948735 +0000 UTC m=+175.375111301" Jul 09 02:02:28.818819 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:02:28.818778 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:03:23.818646 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:03:23.818610 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:03:23.834062 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:03:23.834036 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:03:24.691832 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:03:24.691807 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 02:04:30.484894 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:04:30.484855 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:04:30.485629 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:04:30.485611 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:04:30.497078 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:04:30.497057 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 02:05:52.343658 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.343622 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8"] Jul 09 02:05:52.345726 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.345706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.350506 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.350477 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 02:05:52.350675 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.350659 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 02:05:52.351628 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.351611 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 02:05:52.351981 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.351958 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tjgsb\"" Jul 09 02:05:52.352090 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.351969 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 02:05:52.352994 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.352978 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 02:05:52.377908 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.377877 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8"] Jul 09 02:05:52.491975 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.491938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c92a231d-0ae6-4881-953c-f7ce21aa6c54-cert\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.491975 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.491976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c92a231d-0ae6-4881-953c-f7ce21aa6c54-metrics-cert\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.492231 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.491996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trxp\" (UniqueName: \"kubernetes.io/projected/c92a231d-0ae6-4881-953c-f7ce21aa6c54-kube-api-access-8trxp\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.492231 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.492105 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c92a231d-0ae6-4881-953c-f7ce21aa6c54-manager-config\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.593241 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.593198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c92a231d-0ae6-4881-953c-f7ce21aa6c54-cert\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.593241 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.593241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c92a231d-0ae6-4881-953c-f7ce21aa6c54-metrics-cert\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.593486 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.593289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trxp\" (UniqueName: \"kubernetes.io/projected/c92a231d-0ae6-4881-953c-f7ce21aa6c54-kube-api-access-8trxp\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.593486 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.593336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c92a231d-0ae6-4881-953c-f7ce21aa6c54-manager-config\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.594007 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.593962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c92a231d-0ae6-4881-953c-f7ce21aa6c54-manager-config\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.595803 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.595785 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c92a231d-0ae6-4881-953c-f7ce21aa6c54-metrics-cert\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.595866 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.595821 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c92a231d-0ae6-4881-953c-f7ce21aa6c54-cert\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.605218 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.605185 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trxp\" (UniqueName: \"kubernetes.io/projected/c92a231d-0ae6-4881-953c-f7ce21aa6c54-kube-api-access-8trxp\") pod \"lws-controller-manager-596d76b798-wgcj8\" (UID: \"c92a231d-0ae6-4881-953c-f7ce21aa6c54\") " pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.655356 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.655322 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:52.805363 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.805338 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8"] Jul 09 02:05:52.807771 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:05:52.807745 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92a231d_0ae6_4881_953c_f7ce21aa6c54.slice/crio-fed10d7dc7cba05f5f6b1fcb3334d5e20cce4626a7a3e3dd5b7ffa08c4ee96b2 WatchSource:0}: Error finding container fed10d7dc7cba05f5f6b1fcb3334d5e20cce4626a7a3e3dd5b7ffa08c4ee96b2: Status 404 returned error can't find the container with id fed10d7dc7cba05f5f6b1fcb3334d5e20cce4626a7a3e3dd5b7ffa08c4ee96b2 Jul 09 02:05:52.809780 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:52.809764 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 02:05:53.224331 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:53.224289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" event={"ID":"c92a231d-0ae6-4881-953c-f7ce21aa6c54","Type":"ContainerStarted","Data":"fed10d7dc7cba05f5f6b1fcb3334d5e20cce4626a7a3e3dd5b7ffa08c4ee96b2"} Jul 09 02:05:54.842823 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.842791 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z"] Jul 09 02:05:54.845569 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.845547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:54.848493 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.848466 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 02:05:54.848637 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.848573 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 02:05:54.848942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.848853 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k9pj9\"" Jul 09 02:05:54.849117 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.849104 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 02:05:54.849201 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.849106 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 02:05:54.863002 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.862974 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z"] Jul 09 02:05:54.914821 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.914784 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:54.915022 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.914845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:54.915022 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.914997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-tmp\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:54.915115 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.915063 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srll6\" (UniqueName: \"kubernetes.io/projected/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-kube-api-access-srll6\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:54.915115 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:54.915097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-manifests\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.016140 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.016103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-tmp\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.016350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.016157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srll6\" (UniqueName: \"kubernetes.io/projected/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-kube-api-access-srll6\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.016350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.016176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-manifests\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.016350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.016207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.016350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.016262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.016631 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.016602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-manifests\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.018628 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.018600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.018826 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.018807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-tmp\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.019022 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.019003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.025509 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.025488 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srll6\" (UniqueName: \"kubernetes.io/projected/b64f0a10-3e72-475a-99c3-3e16d42fb9b6-kube-api-access-srll6\") pod \"opendatahub-operator-controller-manager-8687897cbb-d6w6z\" (UID: \"b64f0a10-3e72-475a-99c3-3e16d42fb9b6\") " pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.159549 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.159452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:05:55.235361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.235325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" event={"ID":"c92a231d-0ae6-4881-953c-f7ce21aa6c54","Type":"ContainerStarted","Data":"d093b318b8f7d4f20aead7644790954ec32355c123efd9495e453c82e544589e"} Jul 09 02:05:55.235542 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.235406 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:05:55.253616 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.253562 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" podStartSLOduration=1.038511895 podStartE2EDuration="3.253541387s" totalImagesPullingTime="2.215029492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:05:55.252379321 +0000 UTC m=+385.140541886" watchObservedRunningTime="2026-07-09 02:05:55.253541387 +0000 UTC m=+385.141703951" Jul 09 02:05:55.305314 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:55.305283 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z"] Jul 09 02:05:55.308201 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:05:55.308179 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64f0a10_3e72_475a_99c3_3e16d42fb9b6.slice/crio-5d28c83c452c08d68ccda611112a7e51e616f8632b85b94c65dcf2603e9d3afc WatchSource:0}: Error finding container 5d28c83c452c08d68ccda611112a7e51e616f8632b85b94c65dcf2603e9d3afc: Status 404 returned error can't find the container with id 5d28c83c452c08d68ccda611112a7e51e616f8632b85b94c65dcf2603e9d3afc Jul 09 02:05:56.242516 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:56.242430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" event={"ID":"b64f0a10-3e72-475a-99c3-3e16d42fb9b6","Type":"ContainerStarted","Data":"5d28c83c452c08d68ccda611112a7e51e616f8632b85b94c65dcf2603e9d3afc"} Jul 09 02:05:59.259195 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:05:59.259152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" event={"ID":"b64f0a10-3e72-475a-99c3-3e16d42fb9b6","Type":"ContainerStarted","Data":"cffa95921a6f66d63f86afedadc4dcd84b61153fd9fcc96b62635176cb54ca3a"} Jul 09 02:06:00.269181 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:00.269143 2578 generic.go:356] "Generic (PLEG): container finished" podID="b64f0a10-3e72-475a-99c3-3e16d42fb9b6" containerID="cffa95921a6f66d63f86afedadc4dcd84b61153fd9fcc96b62635176cb54ca3a" exitCode=0 Jul 09 02:06:00.269567 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:00.269223 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" event={"ID":"b64f0a10-3e72-475a-99c3-3e16d42fb9b6","Type":"ContainerDied","Data":"cffa95921a6f66d63f86afedadc4dcd84b61153fd9fcc96b62635176cb54ca3a"} Jul 09 02:06:01.275617 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:01.275583 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" event={"ID":"b64f0a10-3e72-475a-99c3-3e16d42fb9b6","Type":"ContainerStarted","Data":"a836f378bba1f28b82c105a5b17e601dde7cd0ea7161de2877f1c7eab1656ac7"} Jul 09 02:06:01.275998 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:01.275660 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:06:01.298624 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:01.298572 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" podStartSLOduration=2.817692072 podStartE2EDuration="7.298553448s" totalImagesPullingTime="3.411442355s" totalInitContainerRuntime="1.069419021s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:05:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:06:01.296224273 +0000 UTC m=+391.184386838" watchObservedRunningTime="2026-07-09 02:06:01.298553448 +0000 UTC m=+391.186716013" Jul 09 02:06:06.245638 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:06.245610 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-596d76b798-wgcj8" Jul 09 02:06:12.282456 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:12.282415 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8687897cbb-d6w6z" Jul 09 02:06:19.848039 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.847999 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-847fc5468-fkkdf"] Jul 09 02:06:19.861174 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.861134 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-847fc5468-fkkdf"] Jul 09 02:06:19.861381 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.861224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:19.864011 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.863986 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 02:06:19.864161 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.864026 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-l7jkx\"" Jul 09 02:06:19.864161 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.864031 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 02:06:19.929706 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.929671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-tmp\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:19.929878 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.929775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7dx\" (UniqueName: \"kubernetes.io/projected/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-kube-api-access-hd7dx\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:19.929878 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:19.929845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-tls-certs\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:20.030203 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.030165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7dx\" (UniqueName: \"kubernetes.io/projected/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-kube-api-access-hd7dx\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:20.030410 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.030235 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-tls-certs\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:20.030410 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.030291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-tmp\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:20.032660 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.032627 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-tmp\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:20.032864 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.032842 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-tls-certs\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:20.048484 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.048453 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7dx\" (UniqueName: \"kubernetes.io/projected/b8d29b80-fcb4-4919-a72d-cbded5bc5f14-kube-api-access-hd7dx\") pod \"kube-auth-proxy-847fc5468-fkkdf\" (UID: \"b8d29b80-fcb4-4919-a72d-cbded5bc5f14\") " pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:20.171549 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.171455 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" Jul 09 02:06:20.309178 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.309149 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-847fc5468-fkkdf"] Jul 09 02:06:20.311822 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:06:20.311796 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d29b80_fcb4_4919_a72d_cbded5bc5f14.slice/crio-baed6a5022db6c0a996d83c9060c596567d324599a461052ccb44edaeb7ba26c WatchSource:0}: Error finding container baed6a5022db6c0a996d83c9060c596567d324599a461052ccb44edaeb7ba26c: Status 404 returned error can't find the container with id baed6a5022db6c0a996d83c9060c596567d324599a461052ccb44edaeb7ba26c Jul 09 02:06:20.362895 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:20.362862 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" event={"ID":"b8d29b80-fcb4-4919-a72d-cbded5bc5f14","Type":"ContainerStarted","Data":"baed6a5022db6c0a996d83c9060c596567d324599a461052ccb44edaeb7ba26c"} Jul 09 02:06:23.377525 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:23.377482 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" event={"ID":"b8d29b80-fcb4-4919-a72d-cbded5bc5f14","Type":"ContainerStarted","Data":"0840c2b411c03e1eefee1bd87ef7b26a921fb0d16b7eb612693f773c12337f89"} Jul 09 02:06:23.394467 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:06:23.394420 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-847fc5468-fkkdf" podStartSLOduration=1.495604233 podStartE2EDuration="4.394402558s" totalImagesPullingTime="2.898798325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:06:23.393063163 +0000 UTC m=+413.281225729" watchObservedRunningTime="2026-07-09 02:06:23.394402558 +0000 UTC m=+413.282565136" Jul 09 02:07:34.647218 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.647175 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-577b86448b-sr7t6"] Jul 09 02:07:34.650703 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.650677 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.654011 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.653978 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 02:07:34.654216 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.654064 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ldhq7\"" Jul 09 02:07:34.654216 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.654106 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 02:07:34.655391 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.655363 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 02:07:34.656602 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.656583 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 02:07:34.657780 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.657759 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 02:07:34.662410 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.662383 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 02:07:34.664500 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.664479 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577b86448b-sr7t6"] Jul 09 02:07:34.757950 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.757909 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9nh\" (UniqueName: \"kubernetes.io/projected/c9b33f23-24dc-4786-bb15-04a014e3de53-kube-api-access-gz9nh\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.758146 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.757965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b33f23-24dc-4786-bb15-04a014e3de53-console-serving-cert\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.758146 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.758023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-trusted-ca-bundle\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.758146 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.758055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-service-ca\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.758146 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.758079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9b33f23-24dc-4786-bb15-04a014e3de53-console-oauth-config\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.758146 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.758107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-console-config\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.758368 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.758173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-oauth-serving-cert\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.859314 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.859274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9nh\" (UniqueName: \"kubernetes.io/projected/c9b33f23-24dc-4786-bb15-04a014e3de53-kube-api-access-gz9nh\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.859534 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.859327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b33f23-24dc-4786-bb15-04a014e3de53-console-serving-cert\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.859534 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.859354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-trusted-ca-bundle\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.859534 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.859377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-service-ca\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.859534 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.859402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9b33f23-24dc-4786-bb15-04a014e3de53-console-oauth-config\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.859534 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.859444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-console-config\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.859534 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.859498 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-oauth-serving-cert\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.860224 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.860194 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-service-ca\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.860362 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.860327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-oauth-serving-cert\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.860412 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.860378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-trusted-ca-bundle\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.860412 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.860397 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9b33f23-24dc-4786-bb15-04a014e3de53-console-config\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.861880 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.861854 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9b33f23-24dc-4786-bb15-04a014e3de53-console-oauth-config\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.861993 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.861962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b33f23-24dc-4786-bb15-04a014e3de53-console-serving-cert\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.869501 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.869478 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9nh\" (UniqueName: \"kubernetes.io/projected/c9b33f23-24dc-4786-bb15-04a014e3de53-kube-api-access-gz9nh\") pod \"console-577b86448b-sr7t6\" (UID: \"c9b33f23-24dc-4786-bb15-04a014e3de53\") " pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:34.961807 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:34.961773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:35.095721 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:35.095697 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577b86448b-sr7t6"] Jul 09 02:07:35.098126 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:07:35.098102 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b33f23_24dc_4786_bb15_04a014e3de53.slice/crio-8f1ac776b475ed912f4de5890239e0b29749326688844cdf2698968af8bf8976 WatchSource:0}: Error finding container 8f1ac776b475ed912f4de5890239e0b29749326688844cdf2698968af8bf8976: Status 404 returned error can't find the container with id 8f1ac776b475ed912f4de5890239e0b29749326688844cdf2698968af8bf8976 Jul 09 02:07:35.676853 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:35.676817 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577b86448b-sr7t6" event={"ID":"c9b33f23-24dc-4786-bb15-04a014e3de53","Type":"ContainerStarted","Data":"4c375fd20ac06e4bf5173da64ee35b8ea38d4a17149301e615b49943d6849ced"} Jul 09 02:07:35.677242 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:35.676867 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577b86448b-sr7t6" event={"ID":"c9b33f23-24dc-4786-bb15-04a014e3de53","Type":"ContainerStarted","Data":"8f1ac776b475ed912f4de5890239e0b29749326688844cdf2698968af8bf8976"} Jul 09 02:07:35.696233 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:35.696180 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-577b86448b-sr7t6" podStartSLOduration=1.6961488249999999 podStartE2EDuration="1.696148825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:07:35.695092779 +0000 UTC m=+485.583255345" watchObservedRunningTime="2026-07-09 02:07:35.696148825 +0000 UTC m=+485.584311389" Jul 09 02:07:44.962668 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:44.962621 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:44.963173 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:44.962686 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:44.967718 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:44.967691 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:07:45.722370 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:07:45.722337 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-577b86448b-sr7t6" Jul 09 02:08:03.367878 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.367838 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7"] Jul 09 02:08:03.371334 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.371309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.374212 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.374180 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xqpgc\"" Jul 09 02:08:03.374467 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.374300 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 09 02:08:03.374467 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.374445 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 02:08:03.375133 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.375116 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 09 02:08:03.375304 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.375242 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 02:08:03.383033 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.383012 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7"] Jul 09 02:08:03.521455 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.521420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf89t\" (UniqueName: \"kubernetes.io/projected/9d417a88-cff0-4635-93a7-24b03e16f478-kube-api-access-tf89t\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.521455 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.521457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9d417a88-cff0-4635-93a7-24b03e16f478-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.521668 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.521481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d417a88-cff0-4635-93a7-24b03e16f478-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.622391 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.622294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf89t\" (UniqueName: \"kubernetes.io/projected/9d417a88-cff0-4635-93a7-24b03e16f478-kube-api-access-tf89t\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.622391 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.622334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9d417a88-cff0-4635-93a7-24b03e16f478-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.622391 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.622362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d417a88-cff0-4635-93a7-24b03e16f478-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.622994 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.622953 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9d417a88-cff0-4635-93a7-24b03e16f478-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.624757 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.624730 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d417a88-cff0-4635-93a7-24b03e16f478-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.631832 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.631808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf89t\" (UniqueName: \"kubernetes.io/projected/9d417a88-cff0-4635-93a7-24b03e16f478-kube-api-access-tf89t\") pod \"kuadrant-console-plugin-66b5774985-7mwx7\" (UID: \"9d417a88-cff0-4635-93a7-24b03e16f478\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.690844 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.690806 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" Jul 09 02:08:03.827185 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:03.827159 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7"] Jul 09 02:08:03.829463 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:08:03.829433 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d417a88_cff0_4635_93a7_24b03e16f478.slice/crio-4c5837a079273af0d6efea4c2d38df7c47bf247383dfde999e2ddce1f471a343 WatchSource:0}: Error finding container 4c5837a079273af0d6efea4c2d38df7c47bf247383dfde999e2ddce1f471a343: Status 404 returned error can't find the container with id 4c5837a079273af0d6efea4c2d38df7c47bf247383dfde999e2ddce1f471a343 Jul 09 02:08:04.801285 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:04.801218 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" event={"ID":"9d417a88-cff0-4635-93a7-24b03e16f478","Type":"ContainerStarted","Data":"4c5837a079273af0d6efea4c2d38df7c47bf247383dfde999e2ddce1f471a343"} Jul 09 02:08:26.913073 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:26.913040 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" event={"ID":"9d417a88-cff0-4635-93a7-24b03e16f478","Type":"ContainerStarted","Data":"bc0fe7672d359b157610f2eb8f189917aa0ba42aa58916711798c3e08628ef7f"} Jul 09 02:08:26.936107 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:26.936054 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7mwx7" podStartSLOduration=1.440387338 podStartE2EDuration="23.93603741s" totalImagesPullingTime="22.495650072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:08:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:08:26.935229401 +0000 UTC m=+536.823391966" watchObservedRunningTime="2026-07-09 02:08:26.93603741 +0000 UTC m=+536.824199976" Jul 09 02:08:45.023947 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.023909 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vjnwt"] Jul 09 02:08:45.041395 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.041365 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vjnwt"] Jul 09 02:08:45.041553 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.041488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:45.044346 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.044328 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 02:08:45.111312 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.111274 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vjnwt"] Jul 09 02:08:45.195885 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.195845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/edc61f00-5066-43b0-bb11-7b3aa043b321-config-file\") pod \"limitador-limitador-64c94fd64f-vjnwt\" (UID: \"edc61f00-5066-43b0-bb11-7b3aa043b321\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:45.196065 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.195896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7zpw\" (UniqueName: \"kubernetes.io/projected/edc61f00-5066-43b0-bb11-7b3aa043b321-kube-api-access-x7zpw\") pod \"limitador-limitador-64c94fd64f-vjnwt\" (UID: \"edc61f00-5066-43b0-bb11-7b3aa043b321\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:45.297023 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.296923 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/edc61f00-5066-43b0-bb11-7b3aa043b321-config-file\") pod \"limitador-limitador-64c94fd64f-vjnwt\" (UID: \"edc61f00-5066-43b0-bb11-7b3aa043b321\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:45.297023 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.296978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zpw\" (UniqueName: \"kubernetes.io/projected/edc61f00-5066-43b0-bb11-7b3aa043b321-kube-api-access-x7zpw\") pod \"limitador-limitador-64c94fd64f-vjnwt\" (UID: \"edc61f00-5066-43b0-bb11-7b3aa043b321\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:45.297614 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.297594 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/edc61f00-5066-43b0-bb11-7b3aa043b321-config-file\") pod \"limitador-limitador-64c94fd64f-vjnwt\" (UID: \"edc61f00-5066-43b0-bb11-7b3aa043b321\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:45.306158 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.306131 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zpw\" (UniqueName: \"kubernetes.io/projected/edc61f00-5066-43b0-bb11-7b3aa043b321-kube-api-access-x7zpw\") pod \"limitador-limitador-64c94fd64f-vjnwt\" (UID: \"edc61f00-5066-43b0-bb11-7b3aa043b321\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:45.351700 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.351664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:45.488307 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.488277 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vjnwt"] Jul 09 02:08:45.490672 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:08:45.490638 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc61f00_5066_43b0_bb11_7b3aa043b321.slice/crio-dbf3a63d1d9b675d2f6225c98acc015daf712121ce60f510c09d27489d38d99c WatchSource:0}: Error finding container dbf3a63d1d9b675d2f6225c98acc015daf712121ce60f510c09d27489d38d99c: Status 404 returned error can't find the container with id dbf3a63d1d9b675d2f6225c98acc015daf712121ce60f510c09d27489d38d99c Jul 09 02:08:45.806800 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.806763 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-j4gvx"] Jul 09 02:08:45.809721 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.809701 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-j4gvx" Jul 09 02:08:45.812110 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.812088 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-blq99\"" Jul 09 02:08:45.817447 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.817425 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-j4gvx"] Jul 09 02:08:45.901619 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.901580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrb2\" (UniqueName: \"kubernetes.io/projected/e0035e30-1da4-4c77-a5a9-b9698915bcec-kube-api-access-ljrb2\") pod \"authorino-58cd955675-j4gvx\" (UID: \"e0035e30-1da4-4c77-a5a9-b9698915bcec\") " pod="kuadrant-system/authorino-58cd955675-j4gvx" Jul 09 02:08:45.913889 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.913859 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-zclg7"] Jul 09 02:08:45.916168 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.916153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zclg7" Jul 09 02:08:45.925859 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.925835 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zclg7"] Jul 09 02:08:45.996162 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:45.996130 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" event={"ID":"edc61f00-5066-43b0-bb11-7b3aa043b321","Type":"ContainerStarted","Data":"dbf3a63d1d9b675d2f6225c98acc015daf712121ce60f510c09d27489d38d99c"} Jul 09 02:08:46.002557 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.002529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrb2\" (UniqueName: \"kubernetes.io/projected/e0035e30-1da4-4c77-a5a9-b9698915bcec-kube-api-access-ljrb2\") pod \"authorino-58cd955675-j4gvx\" (UID: \"e0035e30-1da4-4c77-a5a9-b9698915bcec\") " pod="kuadrant-system/authorino-58cd955675-j4gvx" Jul 09 02:08:46.002697 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.002609 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwxj2\" (UniqueName: \"kubernetes.io/projected/8166a7cc-3445-440f-ae46-256ea42613cc-kube-api-access-lwxj2\") pod \"authorino-75d556bd77-zclg7\" (UID: \"8166a7cc-3445-440f-ae46-256ea42613cc\") " pod="kuadrant-system/authorino-75d556bd77-zclg7" Jul 09 02:08:46.025333 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.025303 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrb2\" (UniqueName: \"kubernetes.io/projected/e0035e30-1da4-4c77-a5a9-b9698915bcec-kube-api-access-ljrb2\") pod \"authorino-58cd955675-j4gvx\" (UID: \"e0035e30-1da4-4c77-a5a9-b9698915bcec\") " pod="kuadrant-system/authorino-58cd955675-j4gvx" Jul 09 02:08:46.103619 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.103510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwxj2\" (UniqueName: \"kubernetes.io/projected/8166a7cc-3445-440f-ae46-256ea42613cc-kube-api-access-lwxj2\") pod \"authorino-75d556bd77-zclg7\" (UID: \"8166a7cc-3445-440f-ae46-256ea42613cc\") " pod="kuadrant-system/authorino-75d556bd77-zclg7" Jul 09 02:08:46.115417 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.115385 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwxj2\" (UniqueName: \"kubernetes.io/projected/8166a7cc-3445-440f-ae46-256ea42613cc-kube-api-access-lwxj2\") pod \"authorino-75d556bd77-zclg7\" (UID: \"8166a7cc-3445-440f-ae46-256ea42613cc\") " pod="kuadrant-system/authorino-75d556bd77-zclg7" Jul 09 02:08:46.120180 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.120153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-j4gvx" Jul 09 02:08:46.226726 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.226590 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zclg7" Jul 09 02:08:46.302539 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.302279 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-j4gvx"] Jul 09 02:08:46.307087 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:08:46.307052 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0035e30_1da4_4c77_a5a9_b9698915bcec.slice/crio-378915aa997a52edf0226a9e5d9e9094f833c47e9a2e6c8d043b2dbc965adf72 WatchSource:0}: Error finding container 378915aa997a52edf0226a9e5d9e9094f833c47e9a2e6c8d043b2dbc965adf72: Status 404 returned error can't find the container with id 378915aa997a52edf0226a9e5d9e9094f833c47e9a2e6c8d043b2dbc965adf72 Jul 09 02:08:46.439437 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:46.439364 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zclg7"] Jul 09 02:08:46.446081 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:08:46.446046 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8166a7cc_3445_440f_ae46_256ea42613cc.slice/crio-e36ce5194c22ac4842a55dd698e41c035b5f6b05084676c6199c3d9a8f0c0772 WatchSource:0}: Error finding container e36ce5194c22ac4842a55dd698e41c035b5f6b05084676c6199c3d9a8f0c0772: Status 404 returned error can't find the container with id e36ce5194c22ac4842a55dd698e41c035b5f6b05084676c6199c3d9a8f0c0772 Jul 09 02:08:47.010856 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:47.010816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zclg7" event={"ID":"8166a7cc-3445-440f-ae46-256ea42613cc","Type":"ContainerStarted","Data":"e36ce5194c22ac4842a55dd698e41c035b5f6b05084676c6199c3d9a8f0c0772"} Jul 09 02:08:47.014172 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:47.014138 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-j4gvx" event={"ID":"e0035e30-1da4-4c77-a5a9-b9698915bcec","Type":"ContainerStarted","Data":"378915aa997a52edf0226a9e5d9e9094f833c47e9a2e6c8d043b2dbc965adf72"} Jul 09 02:08:51.044438 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:51.044337 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-j4gvx" event={"ID":"e0035e30-1da4-4c77-a5a9-b9698915bcec","Type":"ContainerStarted","Data":"c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762"} Jul 09 02:08:51.045962 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:51.045938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" event={"ID":"edc61f00-5066-43b0-bb11-7b3aa043b321","Type":"ContainerStarted","Data":"deb29f7e7e90d57a9a015941f293974e1ee8d25e4e18cf28752608705bd876e8"} Jul 09 02:08:51.046105 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:51.046013 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:08:51.047401 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:51.047379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zclg7" event={"ID":"8166a7cc-3445-440f-ae46-256ea42613cc","Type":"ContainerStarted","Data":"d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420"} Jul 09 02:08:51.061276 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:51.061210 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-j4gvx" podStartSLOduration=1.665033459 podStartE2EDuration="6.061194588s" totalImagesPullingTime="4.396161129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:08:51.060340696 +0000 UTC m=+560.948503261" watchObservedRunningTime="2026-07-09 02:08:51.061194588 +0000 UTC m=+560.949357221" Jul 09 02:08:51.083049 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:51.082999 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" podStartSLOduration=0.818093901 podStartE2EDuration="6.082980366s" totalImagesPullingTime="5.264886465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:08:51.082055101 +0000 UTC m=+560.970217680" watchObservedRunningTime="2026-07-09 02:08:51.082980366 +0000 UTC m=+560.971142980" Jul 09 02:08:51.097174 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:51.097120 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-zclg7" podStartSLOduration=1.800501758 podStartE2EDuration="6.097104004s" totalImagesPullingTime="4.296602246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:08:51.095463345 +0000 UTC m=+560.983625913" watchObservedRunningTime="2026-07-09 02:08:51.097104004 +0000 UTC m=+560.985266572" Jul 09 02:08:51.122011 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:51.121979 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-j4gvx"] Jul 09 02:08:53.056107 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:53.056046 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-j4gvx" podUID="e0035e30-1da4-4c77-a5a9-b9698915bcec" containerName="authorino" containerID="cri-o://c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762" gracePeriod=30 Jul 09 02:08:53.307646 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:53.307579 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-j4gvx" Jul 09 02:08:53.471328 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:53.471295 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljrb2\" (UniqueName: \"kubernetes.io/projected/e0035e30-1da4-4c77-a5a9-b9698915bcec-kube-api-access-ljrb2\") pod \"e0035e30-1da4-4c77-a5a9-b9698915bcec\" (UID: \"e0035e30-1da4-4c77-a5a9-b9698915bcec\") " Jul 09 02:08:53.473525 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:53.473487 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0035e30-1da4-4c77-a5a9-b9698915bcec-kube-api-access-ljrb2" (OuterVolumeSpecName: "kube-api-access-ljrb2") pod "e0035e30-1da4-4c77-a5a9-b9698915bcec" (UID: "e0035e30-1da4-4c77-a5a9-b9698915bcec"). InnerVolumeSpecName "kube-api-access-ljrb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:08:53.572723 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:53.572632 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrb2\" (UniqueName: \"kubernetes.io/projected/e0035e30-1da4-4c77-a5a9-b9698915bcec-kube-api-access-ljrb2\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:08:54.061379 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.061342 2578 generic.go:356] "Generic (PLEG): container finished" podID="e0035e30-1da4-4c77-a5a9-b9698915bcec" containerID="c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762" exitCode=0 Jul 09 02:08:54.061773 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.061386 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-j4gvx" Jul 09 02:08:54.061773 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.061405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-j4gvx" event={"ID":"e0035e30-1da4-4c77-a5a9-b9698915bcec","Type":"ContainerDied","Data":"c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762"} Jul 09 02:08:54.061773 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.061445 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-j4gvx" event={"ID":"e0035e30-1da4-4c77-a5a9-b9698915bcec","Type":"ContainerDied","Data":"378915aa997a52edf0226a9e5d9e9094f833c47e9a2e6c8d043b2dbc965adf72"} Jul 09 02:08:54.061773 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.061468 2578 scope.go:117] "RemoveContainer" containerID="c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762" Jul 09 02:08:54.069935 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.069914 2578 scope.go:117] "RemoveContainer" containerID="c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762" Jul 09 02:08:54.070225 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:08:54.070207 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762\": container with ID starting with c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762 not found: ID does not exist" containerID="c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762" Jul 09 02:08:54.070302 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.070234 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762"} err="failed to get container status \"c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762\": rpc error: code = NotFound desc = could not find container \"c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762\": container with ID starting with c903884832cd69c10212f84d5829f2ffe999cb72dd515d625a427b9e9cf67762 not found: ID does not exist" Jul 09 02:08:54.089217 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.089187 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-j4gvx"] Jul 09 02:08:54.095313 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.095278 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-j4gvx"] Jul 09 02:08:54.581060 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:08:54.581024 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0035e30-1da4-4c77-a5a9-b9698915bcec" path="/var/lib/kubelet/pods/e0035e30-1da4-4c77-a5a9-b9698915bcec/volumes" Jul 09 02:09:00.304824 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:00.304787 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vjnwt"] Jul 09 02:09:00.305376 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:00.304984 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" podUID="edc61f00-5066-43b0-bb11-7b3aa043b321" containerName="limitador" containerID="cri-o://deb29f7e7e90d57a9a015941f293974e1ee8d25e4e18cf28752608705bd876e8" gracePeriod=30 Jul 09 02:09:00.305734 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:00.305655 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:09:01.093942 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.093900 2578 generic.go:356] "Generic (PLEG): container finished" podID="edc61f00-5066-43b0-bb11-7b3aa043b321" containerID="deb29f7e7e90d57a9a015941f293974e1ee8d25e4e18cf28752608705bd876e8" exitCode=0 Jul 09 02:09:01.094184 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.093975 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" event={"ID":"edc61f00-5066-43b0-bb11-7b3aa043b321","Type":"ContainerDied","Data":"deb29f7e7e90d57a9a015941f293974e1ee8d25e4e18cf28752608705bd876e8"} Jul 09 02:09:01.249065 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.249041 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:09:01.441572 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.441481 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7zpw\" (UniqueName: \"kubernetes.io/projected/edc61f00-5066-43b0-bb11-7b3aa043b321-kube-api-access-x7zpw\") pod \"edc61f00-5066-43b0-bb11-7b3aa043b321\" (UID: \"edc61f00-5066-43b0-bb11-7b3aa043b321\") " Jul 09 02:09:01.441572 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.441566 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/edc61f00-5066-43b0-bb11-7b3aa043b321-config-file\") pod \"edc61f00-5066-43b0-bb11-7b3aa043b321\" (UID: \"edc61f00-5066-43b0-bb11-7b3aa043b321\") " Jul 09 02:09:01.441971 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.441900 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc61f00-5066-43b0-bb11-7b3aa043b321-config-file" (OuterVolumeSpecName: "config-file") pod "edc61f00-5066-43b0-bb11-7b3aa043b321" (UID: "edc61f00-5066-43b0-bb11-7b3aa043b321"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 02:09:01.443622 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.443599 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc61f00-5066-43b0-bb11-7b3aa043b321-kube-api-access-x7zpw" (OuterVolumeSpecName: "kube-api-access-x7zpw") pod "edc61f00-5066-43b0-bb11-7b3aa043b321" (UID: "edc61f00-5066-43b0-bb11-7b3aa043b321"). InnerVolumeSpecName "kube-api-access-x7zpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:09:01.542859 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.542822 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7zpw\" (UniqueName: \"kubernetes.io/projected/edc61f00-5066-43b0-bb11-7b3aa043b321-kube-api-access-x7zpw\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:09:01.542859 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:01.542858 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/edc61f00-5066-43b0-bb11-7b3aa043b321-config-file\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:09:02.099509 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:02.099473 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" event={"ID":"edc61f00-5066-43b0-bb11-7b3aa043b321","Type":"ContainerDied","Data":"dbf3a63d1d9b675d2f6225c98acc015daf712121ce60f510c09d27489d38d99c"} Jul 09 02:09:02.099509 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:02.099500 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vjnwt" Jul 09 02:09:02.099777 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:02.099542 2578 scope.go:117] "RemoveContainer" containerID="deb29f7e7e90d57a9a015941f293974e1ee8d25e4e18cf28752608705bd876e8" Jul 09 02:09:02.122408 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:02.122380 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vjnwt"] Jul 09 02:09:02.126336 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:02.126302 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vjnwt"] Jul 09 02:09:02.580119 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:02.580086 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edc61f00-5066-43b0-bb11-7b3aa043b321" path="/var/lib/kubelet/pods/edc61f00-5066-43b0-bb11-7b3aa043b321/volumes" Jul 09 02:09:06.780158 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.780111 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv"] Jul 09 02:09:06.780652 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.780632 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edc61f00-5066-43b0-bb11-7b3aa043b321" containerName="limitador" Jul 09 02:09:06.780695 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.780657 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="edc61f00-5066-43b0-bb11-7b3aa043b321" containerName="limitador" Jul 09 02:09:06.780729 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.780694 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0035e30-1da4-4c77-a5a9-b9698915bcec" containerName="authorino" Jul 09 02:09:06.780729 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.780703 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0035e30-1da4-4c77-a5a9-b9698915bcec" containerName="authorino" Jul 09 02:09:06.780809 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.780796 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0035e30-1da4-4c77-a5a9-b9698915bcec" containerName="authorino" Jul 09 02:09:06.780843 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.780818 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="edc61f00-5066-43b0-bb11-7b3aa043b321" containerName="limitador" Jul 09 02:09:06.785213 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.785194 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:06.787984 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.787961 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 09 02:09:06.788124 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.788070 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 02:09:06.788124 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.788097 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9sz6s\"" Jul 09 02:09:06.789005 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.788989 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 09 02:09:06.793955 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.793928 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv"] Jul 09 02:09:06.887548 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.887496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9d864215-dcdc-4079-bc73-907b09224c60-data\") pod \"postgres-69fb4594cf-tkmzv\" (UID: \"9d864215-dcdc-4079-bc73-907b09224c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:06.887732 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.887559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w688w\" (UniqueName: \"kubernetes.io/projected/9d864215-dcdc-4079-bc73-907b09224c60-kube-api-access-w688w\") pod \"postgres-69fb4594cf-tkmzv\" (UID: \"9d864215-dcdc-4079-bc73-907b09224c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:06.988299 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.988241 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9d864215-dcdc-4079-bc73-907b09224c60-data\") pod \"postgres-69fb4594cf-tkmzv\" (UID: \"9d864215-dcdc-4079-bc73-907b09224c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:06.988446 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.988313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w688w\" (UniqueName: \"kubernetes.io/projected/9d864215-dcdc-4079-bc73-907b09224c60-kube-api-access-w688w\") pod \"postgres-69fb4594cf-tkmzv\" (UID: \"9d864215-dcdc-4079-bc73-907b09224c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:06.988684 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.988662 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9d864215-dcdc-4079-bc73-907b09224c60-data\") pod \"postgres-69fb4594cf-tkmzv\" (UID: \"9d864215-dcdc-4079-bc73-907b09224c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:06.997607 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:06.997568 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w688w\" (UniqueName: \"kubernetes.io/projected/9d864215-dcdc-4079-bc73-907b09224c60-kube-api-access-w688w\") pod \"postgres-69fb4594cf-tkmzv\" (UID: \"9d864215-dcdc-4079-bc73-907b09224c60\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:07.096294 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:07.096177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:07.235372 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:07.235340 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv"] Jul 09 02:09:07.237351 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:09:07.237323 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d864215_dcdc_4079_bc73_907b09224c60.slice/crio-183e66fbe63dd51a3901016118d836e50f435c91c07fdce60149175d0dedb84e WatchSource:0}: Error finding container 183e66fbe63dd51a3901016118d836e50f435c91c07fdce60149175d0dedb84e: Status 404 returned error can't find the container with id 183e66fbe63dd51a3901016118d836e50f435c91c07fdce60149175d0dedb84e Jul 09 02:09:08.128220 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:08.128177 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" event={"ID":"9d864215-dcdc-4079-bc73-907b09224c60","Type":"ContainerStarted","Data":"183e66fbe63dd51a3901016118d836e50f435c91c07fdce60149175d0dedb84e"} Jul 09 02:09:14.160038 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:14.159998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" event={"ID":"9d864215-dcdc-4079-bc73-907b09224c60","Type":"ContainerStarted","Data":"12424846435e4e4adff9e953fc71a98cc21453eda29b3ddb7ff9ffd621f31595"} Jul 09 02:09:14.160467 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:14.160114 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:14.181097 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:14.181040 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" podStartSLOduration=2.077959162 podStartE2EDuration="8.181018287s" totalImagesPullingTime="6.103059125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:09:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:09:14.179498451 +0000 UTC m=+584.067661027" watchObservedRunningTime="2026-07-09 02:09:14.181018287 +0000 UTC m=+584.069180852" Jul 09 02:09:20.195042 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:20.195010 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-tkmzv" Jul 09 02:09:20.735015 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:20.734984 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6b7d5"] Jul 09 02:09:20.801166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:20.801128 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6b7d5"] Jul 09 02:09:20.801341 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:20.801278 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6b7d5" Jul 09 02:09:20.917301 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:20.917242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2nwr\" (UniqueName: \"kubernetes.io/projected/f521fdb0-c7fd-40a8-981c-c80f05931e01-kube-api-access-h2nwr\") pod \"authorino-5d4b67c446-6b7d5\" (UID: \"f521fdb0-c7fd-40a8-981c-c80f05931e01\") " pod="kuadrant-system/authorino-5d4b67c446-6b7d5" Jul 09 02:09:20.995067 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:20.994979 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6b7d5"] Jul 09 02:09:20.995294 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:09:20.995268 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-h2nwr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-6b7d5" podUID="f521fdb0-c7fd-40a8-981c-c80f05931e01" Jul 09 02:09:21.018372 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.018342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2nwr\" (UniqueName: \"kubernetes.io/projected/f521fdb0-c7fd-40a8-981c-c80f05931e01-kube-api-access-h2nwr\") pod \"authorino-5d4b67c446-6b7d5\" (UID: \"f521fdb0-c7fd-40a8-981c-c80f05931e01\") " pod="kuadrant-system/authorino-5d4b67c446-6b7d5" Jul 09 02:09:21.024719 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.024691 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5574668dd8-xr9r8"] Jul 09 02:09:21.027810 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.027789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2nwr\" (UniqueName: \"kubernetes.io/projected/f521fdb0-c7fd-40a8-981c-c80f05931e01-kube-api-access-h2nwr\") pod \"authorino-5d4b67c446-6b7d5\" (UID: \"f521fdb0-c7fd-40a8-981c-c80f05931e01\") " pod="kuadrant-system/authorino-5d4b67c446-6b7d5" Jul 09 02:09:21.041902 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.041872 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5574668dd8-xr9r8"] Jul 09 02:09:21.042046 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.041980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5574668dd8-xr9r8" Jul 09 02:09:21.118850 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.118817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqwc\" (UniqueName: \"kubernetes.io/projected/51a02800-8011-46d7-a67c-d2eef1844f12-kube-api-access-vqqwc\") pod \"authorino-5574668dd8-xr9r8\" (UID: \"51a02800-8011-46d7-a67c-d2eef1844f12\") " pod="kuadrant-system/authorino-5574668dd8-xr9r8" Jul 09 02:09:21.192927 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.192895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6b7d5" Jul 09 02:09:21.197880 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.197847 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5574668dd8-xr9r8"] Jul 09 02:09:21.198265 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:09:21.198105 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vqqwc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5574668dd8-xr9r8" podUID="51a02800-8011-46d7-a67c-d2eef1844f12" Jul 09 02:09:21.213017 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.212991 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6b7d5" Jul 09 02:09:21.219504 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.219475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqwc\" (UniqueName: \"kubernetes.io/projected/51a02800-8011-46d7-a67c-d2eef1844f12-kube-api-access-vqqwc\") pod \"authorino-5574668dd8-xr9r8\" (UID: \"51a02800-8011-46d7-a67c-d2eef1844f12\") " pod="kuadrant-system/authorino-5574668dd8-xr9r8" Jul 09 02:09:21.223590 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.223563 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74b96dc968-ps6hz"] Jul 09 02:09:21.230217 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.230197 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqwc\" (UniqueName: \"kubernetes.io/projected/51a02800-8011-46d7-a67c-d2eef1844f12-kube-api-access-vqqwc\") pod \"authorino-5574668dd8-xr9r8\" (UID: \"51a02800-8011-46d7-a67c-d2eef1844f12\") " pod="kuadrant-system/authorino-5574668dd8-xr9r8" Jul 09 02:09:21.235611 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.235589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:09:21.236061 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.236042 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74b96dc968-ps6hz"] Jul 09 02:09:21.238799 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.238777 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 09 02:09:21.320783 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.320704 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2nwr\" (UniqueName: \"kubernetes.io/projected/f521fdb0-c7fd-40a8-981c-c80f05931e01-kube-api-access-h2nwr\") pod \"f521fdb0-c7fd-40a8-981c-c80f05931e01\" (UID: \"f521fdb0-c7fd-40a8-981c-c80f05931e01\") " Jul 09 02:09:21.320929 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.320849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-588hz\" (UniqueName: \"kubernetes.io/projected/106900b8-5ef6-4a01-9967-4afeab9ae183-kube-api-access-588hz\") pod \"authorino-74b96dc968-ps6hz\" (UID: \"106900b8-5ef6-4a01-9967-4afeab9ae183\") " pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:09:21.320929 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.320900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/106900b8-5ef6-4a01-9967-4afeab9ae183-tls-cert\") pod \"authorino-74b96dc968-ps6hz\" (UID: \"106900b8-5ef6-4a01-9967-4afeab9ae183\") " pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:09:21.322698 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.322673 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f521fdb0-c7fd-40a8-981c-c80f05931e01-kube-api-access-h2nwr" (OuterVolumeSpecName: "kube-api-access-h2nwr") pod "f521fdb0-c7fd-40a8-981c-c80f05931e01" (UID: "f521fdb0-c7fd-40a8-981c-c80f05931e01"). InnerVolumeSpecName "kube-api-access-h2nwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:09:21.422195 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.422159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-588hz\" (UniqueName: \"kubernetes.io/projected/106900b8-5ef6-4a01-9967-4afeab9ae183-kube-api-access-588hz\") pod \"authorino-74b96dc968-ps6hz\" (UID: \"106900b8-5ef6-4a01-9967-4afeab9ae183\") " pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:09:21.422404 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.422214 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/106900b8-5ef6-4a01-9967-4afeab9ae183-tls-cert\") pod \"authorino-74b96dc968-ps6hz\" (UID: \"106900b8-5ef6-4a01-9967-4afeab9ae183\") " pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:09:21.422404 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.422316 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2nwr\" (UniqueName: \"kubernetes.io/projected/f521fdb0-c7fd-40a8-981c-c80f05931e01-kube-api-access-h2nwr\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:09:21.424681 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.424658 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/106900b8-5ef6-4a01-9967-4afeab9ae183-tls-cert\") pod \"authorino-74b96dc968-ps6hz\" (UID: \"106900b8-5ef6-4a01-9967-4afeab9ae183\") " pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:09:21.430984 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.430951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-588hz\" (UniqueName: \"kubernetes.io/projected/106900b8-5ef6-4a01-9967-4afeab9ae183-kube-api-access-588hz\") pod \"authorino-74b96dc968-ps6hz\" (UID: \"106900b8-5ef6-4a01-9967-4afeab9ae183\") " pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:09:21.546885 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.546841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:09:21.694374 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:21.694349 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74b96dc968-ps6hz"] Jul 09 02:09:21.696666 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:09:21.696644 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod106900b8_5ef6_4a01_9967_4afeab9ae183.slice/crio-58ca7dd06124d823b9b84c79639511de82206a1a4957ffbd44ec57a1710acb58 WatchSource:0}: Error finding container 58ca7dd06124d823b9b84c79639511de82206a1a4957ffbd44ec57a1710acb58: Status 404 returned error can't find the container with id 58ca7dd06124d823b9b84c79639511de82206a1a4957ffbd44ec57a1710acb58 Jul 09 02:09:22.200233 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.200205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-6b7d5" Jul 09 02:09:22.200233 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.200227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5574668dd8-xr9r8" Jul 09 02:09:22.200766 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.200195 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b96dc968-ps6hz" event={"ID":"106900b8-5ef6-4a01-9967-4afeab9ae183","Type":"ContainerStarted","Data":"58ca7dd06124d823b9b84c79639511de82206a1a4957ffbd44ec57a1710acb58"} Jul 09 02:09:22.234583 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.234544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5574668dd8-xr9r8" Jul 09 02:09:22.254900 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.254869 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6b7d5"] Jul 09 02:09:22.259366 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.259333 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-6b7d5"] Jul 09 02:09:22.331060 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.331026 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqqwc\" (UniqueName: \"kubernetes.io/projected/51a02800-8011-46d7-a67c-d2eef1844f12-kube-api-access-vqqwc\") pod \"51a02800-8011-46d7-a67c-d2eef1844f12\" (UID: \"51a02800-8011-46d7-a67c-d2eef1844f12\") " Jul 09 02:09:22.333127 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.333099 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a02800-8011-46d7-a67c-d2eef1844f12-kube-api-access-vqqwc" (OuterVolumeSpecName: "kube-api-access-vqqwc") pod "51a02800-8011-46d7-a67c-d2eef1844f12" (UID: "51a02800-8011-46d7-a67c-d2eef1844f12"). InnerVolumeSpecName "kube-api-access-vqqwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:09:22.432346 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.432304 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqwc\" (UniqueName: \"kubernetes.io/projected/51a02800-8011-46d7-a67c-d2eef1844f12-kube-api-access-vqqwc\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:09:22.580183 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:22.580149 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f521fdb0-c7fd-40a8-981c-c80f05931e01" path="/var/lib/kubelet/pods/f521fdb0-c7fd-40a8-981c-c80f05931e01/volumes" Jul 09 02:09:23.206476 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.206441 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b96dc968-ps6hz" event={"ID":"106900b8-5ef6-4a01-9967-4afeab9ae183","Type":"ContainerStarted","Data":"76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1"} Jul 09 02:09:23.206476 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.206489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5574668dd8-xr9r8" Jul 09 02:09:23.222897 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.222845 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74b96dc968-ps6hz" podStartSLOduration=1.5579659989999999 podStartE2EDuration="2.222828853s" totalImagesPullingTime="664.862854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:09:23.221696561 +0000 UTC m=+593.109859127" watchObservedRunningTime="2026-07-09 02:09:23.222828853 +0000 UTC m=+593.110991418" Jul 09 02:09:23.247405 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.247374 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5574668dd8-xr9r8"] Jul 09 02:09:23.252355 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.252327 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zclg7"] Jul 09 02:09:23.252522 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.252500 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-zclg7" podUID="8166a7cc-3445-440f-ae46-256ea42613cc" containerName="authorino" containerID="cri-o://d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420" gracePeriod=30 Jul 09 02:09:23.255658 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.255630 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5574668dd8-xr9r8"] Jul 09 02:09:23.500638 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.500615 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zclg7" Jul 09 02:09:23.643397 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.643362 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwxj2\" (UniqueName: \"kubernetes.io/projected/8166a7cc-3445-440f-ae46-256ea42613cc-kube-api-access-lwxj2\") pod \"8166a7cc-3445-440f-ae46-256ea42613cc\" (UID: \"8166a7cc-3445-440f-ae46-256ea42613cc\") " Jul 09 02:09:23.645518 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.645491 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8166a7cc-3445-440f-ae46-256ea42613cc-kube-api-access-lwxj2" (OuterVolumeSpecName: "kube-api-access-lwxj2") pod "8166a7cc-3445-440f-ae46-256ea42613cc" (UID: "8166a7cc-3445-440f-ae46-256ea42613cc"). InnerVolumeSpecName "kube-api-access-lwxj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:09:23.744265 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:23.744145 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwxj2\" (UniqueName: \"kubernetes.io/projected/8166a7cc-3445-440f-ae46-256ea42613cc-kube-api-access-lwxj2\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:09:24.212231 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.212195 2578 generic.go:356] "Generic (PLEG): container finished" podID="8166a7cc-3445-440f-ae46-256ea42613cc" containerID="d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420" exitCode=0 Jul 09 02:09:24.212716 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.212241 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zclg7" Jul 09 02:09:24.212716 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.212286 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zclg7" event={"ID":"8166a7cc-3445-440f-ae46-256ea42613cc","Type":"ContainerDied","Data":"d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420"} Jul 09 02:09:24.212716 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.212333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zclg7" event={"ID":"8166a7cc-3445-440f-ae46-256ea42613cc","Type":"ContainerDied","Data":"e36ce5194c22ac4842a55dd698e41c035b5f6b05084676c6199c3d9a8f0c0772"} Jul 09 02:09:24.212716 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.212354 2578 scope.go:117] "RemoveContainer" containerID="d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420" Jul 09 02:09:24.220742 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.220724 2578 scope.go:117] "RemoveContainer" containerID="d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420" Jul 09 02:09:24.221017 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:09:24.221001 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420\": container with ID starting with d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420 not found: ID does not exist" containerID="d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420" Jul 09 02:09:24.221092 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.221025 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420"} err="failed to get container status \"d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420\": rpc error: code = NotFound desc = could not find container \"d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420\": container with ID starting with d6f3bd000039613815eca495575bb374cfa6eb29b7d0934e69adf89af7ffb420 not found: ID does not exist" Jul 09 02:09:24.241855 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.237983 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zclg7"] Jul 09 02:09:24.245026 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.244999 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zclg7"] Jul 09 02:09:24.579707 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.579626 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51a02800-8011-46d7-a67c-d2eef1844f12" path="/var/lib/kubelet/pods/51a02800-8011-46d7-a67c-d2eef1844f12/volumes" Jul 09 02:09:24.579863 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:24.579852 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8166a7cc-3445-440f-ae46-256ea42613cc" path="/var/lib/kubelet/pods/8166a7cc-3445-440f-ae46-256ea42613cc/volumes" Jul 09 02:09:30.513941 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:30.513908 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:09:30.515271 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:09:30.515236 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:10:05.112301 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.112203 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-t2kp9"] Jul 09 02:10:05.112758 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.112575 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8166a7cc-3445-440f-ae46-256ea42613cc" containerName="authorino" Jul 09 02:10:05.112758 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.112588 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8166a7cc-3445-440f-ae46-256ea42613cc" containerName="authorino" Jul 09 02:10:05.112758 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.112647 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8166a7cc-3445-440f-ae46-256ea42613cc" containerName="authorino" Jul 09 02:10:05.115417 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.115401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:05.118134 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.118114 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 09 02:10:05.118257 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.118159 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ktthp\"" Jul 09 02:10:05.127717 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.127692 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-t2kp9"] Jul 09 02:10:05.194512 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.194477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqc27\" (UniqueName: \"kubernetes.io/projected/d3ce3fb6-b789-40a9-b139-6f0dbc840523-kube-api-access-xqc27\") pod \"payload-processing-58f86c7798-t2kp9\" (UID: \"d3ce3fb6-b789-40a9-b139-6f0dbc840523\") " pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:05.194684 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.194546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ce3fb6-b789-40a9-b139-6f0dbc840523-plugins-config-volume\") pod \"payload-processing-58f86c7798-t2kp9\" (UID: \"d3ce3fb6-b789-40a9-b139-6f0dbc840523\") " pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:05.295838 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.295785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ce3fb6-b789-40a9-b139-6f0dbc840523-plugins-config-volume\") pod \"payload-processing-58f86c7798-t2kp9\" (UID: \"d3ce3fb6-b789-40a9-b139-6f0dbc840523\") " pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:05.296017 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.295913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqc27\" (UniqueName: \"kubernetes.io/projected/d3ce3fb6-b789-40a9-b139-6f0dbc840523-kube-api-access-xqc27\") pod \"payload-processing-58f86c7798-t2kp9\" (UID: \"d3ce3fb6-b789-40a9-b139-6f0dbc840523\") " pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:05.296622 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.296601 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3ce3fb6-b789-40a9-b139-6f0dbc840523-plugins-config-volume\") pod \"payload-processing-58f86c7798-t2kp9\" (UID: \"d3ce3fb6-b789-40a9-b139-6f0dbc840523\") " pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:05.317384 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.317351 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqc27\" (UniqueName: \"kubernetes.io/projected/d3ce3fb6-b789-40a9-b139-6f0dbc840523-kube-api-access-xqc27\") pod \"payload-processing-58f86c7798-t2kp9\" (UID: \"d3ce3fb6-b789-40a9-b139-6f0dbc840523\") " pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:05.425631 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.425528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:05.571024 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:05.570905 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-t2kp9"] Jul 09 02:10:05.573786 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:10:05.573757 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ce3fb6_b789_40a9_b139_6f0dbc840523.slice/crio-a0dadad76c1b76dc82349a26cff9a104d195a5044843b868a20fc7345cdf2643 WatchSource:0}: Error finding container a0dadad76c1b76dc82349a26cff9a104d195a5044843b868a20fc7345cdf2643: Status 404 returned error can't find the container with id a0dadad76c1b76dc82349a26cff9a104d195a5044843b868a20fc7345cdf2643 Jul 09 02:10:06.395584 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:06.395548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" event={"ID":"d3ce3fb6-b789-40a9-b139-6f0dbc840523","Type":"ContainerStarted","Data":"a0dadad76c1b76dc82349a26cff9a104d195a5044843b868a20fc7345cdf2643"} Jul 09 02:10:07.400966 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:07.400869 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" event={"ID":"d3ce3fb6-b789-40a9-b139-6f0dbc840523","Type":"ContainerStarted","Data":"06d81d0d166479bdf819cdd8a5fb74cdb312f8f70bde77091f4590013f712063"} Jul 09 02:10:07.401363 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:07.400975 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:07.420423 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:07.420374 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" podStartSLOduration=0.980615921 podStartE2EDuration="2.420353973s" totalImagesPullingTime="1.439738052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:10:07.418929819 +0000 UTC m=+637.307092385" watchObservedRunningTime="2026-07-09 02:10:07.420353973 +0000 UTC m=+637.308516543" Jul 09 02:10:18.407884 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:18.407855 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-t2kp9" Jul 09 02:10:19.703898 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.703864 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6"] Jul 09 02:10:19.707206 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.707185 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.710076 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.710052 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 02:10:19.710169 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.710095 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tcfth\"" Jul 09 02:10:19.711045 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.711026 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 02:10:19.711098 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.711041 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 02:10:19.720104 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.720079 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6"] Jul 09 02:10:19.832036 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.831995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.832207 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.832049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjlx7\" (UniqueName: \"kubernetes.io/projected/a463e3c0-2ac4-41e1-8f83-45164d4e1702-kube-api-access-vjlx7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.832207 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.832135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.832207 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.832184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a463e3c0-2ac4-41e1-8f83-45164d4e1702-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.832341 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.832210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.832341 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.832227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.933679 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.933641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlx7\" (UniqueName: \"kubernetes.io/projected/a463e3c0-2ac4-41e1-8f83-45164d4e1702-kube-api-access-vjlx7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.933858 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.933687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.933858 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.933729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a463e3c0-2ac4-41e1-8f83-45164d4e1702-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.933858 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.933759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.933858 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.933781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.933858 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.933855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.934220 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.934191 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.934325 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.934214 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.934325 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.934240 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.936138 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.936115 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a463e3c0-2ac4-41e1-8f83-45164d4e1702-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.936336 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.936319 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a463e3c0-2ac4-41e1-8f83-45164d4e1702-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:19.944968 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:19.944942 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlx7\" (UniqueName: \"kubernetes.io/projected/a463e3c0-2ac4-41e1-8f83-45164d4e1702-kube-api-access-vjlx7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6\" (UID: \"a463e3c0-2ac4-41e1-8f83-45164d4e1702\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:20.017761 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:20.017665 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:20.158773 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:20.158744 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6"] Jul 09 02:10:20.160663 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:10:20.160638 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda463e3c0_2ac4_41e1_8f83_45164d4e1702.slice/crio-2f3ac426ec18de98bf70375d77088d2fb622794c253e206cc437f5bb47a8016a WatchSource:0}: Error finding container 2f3ac426ec18de98bf70375d77088d2fb622794c253e206cc437f5bb47a8016a: Status 404 returned error can't find the container with id 2f3ac426ec18de98bf70375d77088d2fb622794c253e206cc437f5bb47a8016a Jul 09 02:10:20.464807 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:20.464523 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" event={"ID":"a463e3c0-2ac4-41e1-8f83-45164d4e1702","Type":"ContainerStarted","Data":"2f3ac426ec18de98bf70375d77088d2fb622794c253e206cc437f5bb47a8016a"} Jul 09 02:10:21.470213 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:21.470175 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" event={"ID":"a463e3c0-2ac4-41e1-8f83-45164d4e1702","Type":"ContainerStarted","Data":"9fe5bf0b77c6156d58c8a3fc55f738d1c71fc18f990825a06c86e755d7d21e43"} Jul 09 02:10:21.493746 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:21.493693 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" podStartSLOduration=1.33474145 podStartE2EDuration="2.493674134s" totalImagesPullingTime="1.158932684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:10:21.491779023 +0000 UTC m=+651.379941587" watchObservedRunningTime="2026-07-09 02:10:21.493674134 +0000 UTC m=+651.381836750" Jul 09 02:10:30.018391 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:30.018346 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:30.018889 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:30.018421 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:30.031146 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:30.031118 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:30.523003 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:30.522970 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6" Jul 09 02:10:37.312321 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.312270 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8"] Jul 09 02:10:37.317408 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.317388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.320938 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.320910 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 09 02:10:37.331972 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.331942 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8"] Jul 09 02:10:37.388067 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.388033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.388067 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.388076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3146dcad-7a54-40d3-88d7-1cc853cb02c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.388350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.388102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.388350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.388173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.388350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.388190 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpdw\" (UniqueName: \"kubernetes.io/projected/3146dcad-7a54-40d3-88d7-1cc853cb02c8-kube-api-access-ffpdw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.388350 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.388268 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489409 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489409 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpdw\" (UniqueName: \"kubernetes.io/projected/3146dcad-7a54-40d3-88d7-1cc853cb02c8-kube-api-access-ffpdw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489667 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489667 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489667 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3146dcad-7a54-40d3-88d7-1cc853cb02c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489667 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489905 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489879 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489965 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489926 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.489965 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.489958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.491793 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.491770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3146dcad-7a54-40d3-88d7-1cc853cb02c8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.491989 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.491970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3146dcad-7a54-40d3-88d7-1cc853cb02c8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.500342 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.500318 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpdw\" (UniqueName: \"kubernetes.io/projected/3146dcad-7a54-40d3-88d7-1cc853cb02c8-kube-api-access-ffpdw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9ddr8\" (UID: \"3146dcad-7a54-40d3-88d7-1cc853cb02c8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.627520 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.627419 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:37.766210 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:37.766094 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8"] Jul 09 02:10:37.768506 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:10:37.768477 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3146dcad_7a54_40d3_88d7_1cc853cb02c8.slice/crio-a23d1bc383f26f14dc9c59a4695fe7a689f0a8abd70c59112122868ddaf64be5 WatchSource:0}: Error finding container a23d1bc383f26f14dc9c59a4695fe7a689f0a8abd70c59112122868ddaf64be5: Status 404 returned error can't find the container with id a23d1bc383f26f14dc9c59a4695fe7a689f0a8abd70c59112122868ddaf64be5 Jul 09 02:10:38.550879 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:38.550759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" event={"ID":"3146dcad-7a54-40d3-88d7-1cc853cb02c8","Type":"ContainerStarted","Data":"7e4eef2157653365e8031393d7a723b81166e14945c12c34e1a16f5f65445e91"} Jul 09 02:10:38.550879 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:38.550823 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" event={"ID":"3146dcad-7a54-40d3-88d7-1cc853cb02c8","Type":"ContainerStarted","Data":"a23d1bc383f26f14dc9c59a4695fe7a689f0a8abd70c59112122868ddaf64be5"} Jul 09 02:10:38.579324 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:38.579242 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" podStartSLOduration=1.368542593 podStartE2EDuration="1.579225143s" totalImagesPullingTime="210.68255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:10:38.57903791 +0000 UTC m=+668.467200477" watchObservedRunningTime="2026-07-09 02:10:38.579225143 +0000 UTC m=+668.467387708" Jul 09 02:10:47.628505 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:47.628465 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:47.628950 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:47.628538 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:47.640781 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:47.640749 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:10:48.607233 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:10:48.607201 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9ddr8" Jul 09 02:12:20.010365 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.010336 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c5bd64b69-dspdm"] Jul 09 02:12:20.013596 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.013581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c5bd64b69-dspdm" Jul 09 02:12:20.025654 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.025628 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c5bd64b69-dspdm"] Jul 09 02:12:20.111206 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.111169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f5d397f-b8bd-452e-a4a2-3338c893dc2a-tls-cert\") pod \"authorino-6c5bd64b69-dspdm\" (UID: \"5f5d397f-b8bd-452e-a4a2-3338c893dc2a\") " pod="kuadrant-system/authorino-6c5bd64b69-dspdm" Jul 09 02:12:20.111408 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.111227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6b5v\" (UniqueName: \"kubernetes.io/projected/5f5d397f-b8bd-452e-a4a2-3338c893dc2a-kube-api-access-w6b5v\") pod \"authorino-6c5bd64b69-dspdm\" (UID: \"5f5d397f-b8bd-452e-a4a2-3338c893dc2a\") " pod="kuadrant-system/authorino-6c5bd64b69-dspdm" Jul 09 02:12:20.212141 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.212104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f5d397f-b8bd-452e-a4a2-3338c893dc2a-tls-cert\") pod \"authorino-6c5bd64b69-dspdm\" (UID: \"5f5d397f-b8bd-452e-a4a2-3338c893dc2a\") " pod="kuadrant-system/authorino-6c5bd64b69-dspdm" Jul 09 02:12:20.212357 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.212166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6b5v\" (UniqueName: \"kubernetes.io/projected/5f5d397f-b8bd-452e-a4a2-3338c893dc2a-kube-api-access-w6b5v\") pod \"authorino-6c5bd64b69-dspdm\" (UID: \"5f5d397f-b8bd-452e-a4a2-3338c893dc2a\") " pod="kuadrant-system/authorino-6c5bd64b69-dspdm" Jul 09 02:12:20.214649 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.214618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f5d397f-b8bd-452e-a4a2-3338c893dc2a-tls-cert\") pod \"authorino-6c5bd64b69-dspdm\" (UID: \"5f5d397f-b8bd-452e-a4a2-3338c893dc2a\") " pod="kuadrant-system/authorino-6c5bd64b69-dspdm" Jul 09 02:12:20.223042 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.223019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6b5v\" (UniqueName: \"kubernetes.io/projected/5f5d397f-b8bd-452e-a4a2-3338c893dc2a-kube-api-access-w6b5v\") pod \"authorino-6c5bd64b69-dspdm\" (UID: \"5f5d397f-b8bd-452e-a4a2-3338c893dc2a\") " pod="kuadrant-system/authorino-6c5bd64b69-dspdm" Jul 09 02:12:20.322000 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.321905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c5bd64b69-dspdm" Jul 09 02:12:20.471317 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.471288 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c5bd64b69-dspdm"] Jul 09 02:12:20.473021 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:12:20.472998 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f5d397f_b8bd_452e_a4a2_3338c893dc2a.slice/crio-71497e9b9eeb5654b54a6ac78226cd6cb73d5f687ceec0f1b3b270c37bc3bcac WatchSource:0}: Error finding container 71497e9b9eeb5654b54a6ac78226cd6cb73d5f687ceec0f1b3b270c37bc3bcac: Status 404 returned error can't find the container with id 71497e9b9eeb5654b54a6ac78226cd6cb73d5f687ceec0f1b3b270c37bc3bcac Jul 09 02:12:20.474349 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:20.474330 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 02:12:21.017684 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.017641 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c5bd64b69-dspdm" event={"ID":"5f5d397f-b8bd-452e-a4a2-3338c893dc2a","Type":"ContainerStarted","Data":"950690a45e85c3f551e8aff1cdcce26a752575b68d579b0cc57cea0f9cc64cce"} Jul 09 02:12:21.018192 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.017696 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c5bd64b69-dspdm" event={"ID":"5f5d397f-b8bd-452e-a4a2-3338c893dc2a","Type":"ContainerStarted","Data":"71497e9b9eeb5654b54a6ac78226cd6cb73d5f687ceec0f1b3b270c37bc3bcac"} Jul 09 02:12:21.035211 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.035154 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c5bd64b69-dspdm" podStartSLOduration=1.632267948 podStartE2EDuration="2.035138049s" totalImagesPullingTime="402.870101ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:12:21.034070251 +0000 UTC m=+770.922232817" watchObservedRunningTime="2026-07-09 02:12:21.035138049 +0000 UTC m=+770.923300613" Jul 09 02:12:21.074860 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.074828 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74b96dc968-ps6hz"] Jul 09 02:12:21.075070 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.075043 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-74b96dc968-ps6hz" podUID="106900b8-5ef6-4a01-9967-4afeab9ae183" containerName="authorino" containerID="cri-o://76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1" gracePeriod=30 Jul 09 02:12:21.341191 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.341167 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:12:21.424179 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.424140 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/106900b8-5ef6-4a01-9967-4afeab9ae183-tls-cert\") pod \"106900b8-5ef6-4a01-9967-4afeab9ae183\" (UID: \"106900b8-5ef6-4a01-9967-4afeab9ae183\") " Jul 09 02:12:21.424391 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.424189 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-588hz\" (UniqueName: \"kubernetes.io/projected/106900b8-5ef6-4a01-9967-4afeab9ae183-kube-api-access-588hz\") pod \"106900b8-5ef6-4a01-9967-4afeab9ae183\" (UID: \"106900b8-5ef6-4a01-9967-4afeab9ae183\") " Jul 09 02:12:21.426283 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.426237 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106900b8-5ef6-4a01-9967-4afeab9ae183-kube-api-access-588hz" (OuterVolumeSpecName: "kube-api-access-588hz") pod "106900b8-5ef6-4a01-9967-4afeab9ae183" (UID: "106900b8-5ef6-4a01-9967-4afeab9ae183"). InnerVolumeSpecName "kube-api-access-588hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:12:21.434005 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.433973 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106900b8-5ef6-4a01-9967-4afeab9ae183-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "106900b8-5ef6-4a01-9967-4afeab9ae183" (UID: "106900b8-5ef6-4a01-9967-4afeab9ae183"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:12:21.525724 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.525685 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/106900b8-5ef6-4a01-9967-4afeab9ae183-tls-cert\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:12:21.525724 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:21.525721 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-588hz\" (UniqueName: \"kubernetes.io/projected/106900b8-5ef6-4a01-9967-4afeab9ae183-kube-api-access-588hz\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:12:22.023902 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.023872 2578 generic.go:356] "Generic (PLEG): container finished" podID="106900b8-5ef6-4a01-9967-4afeab9ae183" containerID="76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1" exitCode=0 Jul 09 02:12:22.024374 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.023916 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74b96dc968-ps6hz" Jul 09 02:12:22.024374 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.023960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b96dc968-ps6hz" event={"ID":"106900b8-5ef6-4a01-9967-4afeab9ae183","Type":"ContainerDied","Data":"76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1"} Jul 09 02:12:22.024374 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.024006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74b96dc968-ps6hz" event={"ID":"106900b8-5ef6-4a01-9967-4afeab9ae183","Type":"ContainerDied","Data":"58ca7dd06124d823b9b84c79639511de82206a1a4957ffbd44ec57a1710acb58"} Jul 09 02:12:22.024374 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.024022 2578 scope.go:117] "RemoveContainer" containerID="76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1" Jul 09 02:12:22.033031 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.033012 2578 scope.go:117] "RemoveContainer" containerID="76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1" Jul 09 02:12:22.033347 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:12:22.033330 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1\": container with ID starting with 76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1 not found: ID does not exist" containerID="76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1" Jul 09 02:12:22.033413 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.033355 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1"} err="failed to get container status \"76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1\": rpc error: code = NotFound desc = could not find container \"76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1\": container with ID starting with 76602b820d0f32fe2a336882e044d89a1de5abf12e7cbdb8aecf28add95ca6b1 not found: ID does not exist" Jul 09 02:12:22.046861 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.046834 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74b96dc968-ps6hz"] Jul 09 02:12:22.053053 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.053026 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74b96dc968-ps6hz"] Jul 09 02:12:22.580822 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:12:22.580788 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="106900b8-5ef6-4a01-9967-4afeab9ae183" path="/var/lib/kubelet/pods/106900b8-5ef6-4a01-9967-4afeab9ae183/volumes" Jul 09 02:14:30.550595 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:14:30.550564 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:14:30.552600 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:14:30.552577 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:15:00.136301 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.136261 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz"] Jul 09 02:15:00.136704 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.136607 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="106900b8-5ef6-4a01-9967-4afeab9ae183" containerName="authorino" Jul 09 02:15:00.136704 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.136618 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="106900b8-5ef6-4a01-9967-4afeab9ae183" containerName="authorino" Jul 09 02:15:00.136704 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.136679 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="106900b8-5ef6-4a01-9967-4afeab9ae183" containerName="authorino" Jul 09 02:15:00.139684 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.139661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" Jul 09 02:15:00.142343 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.142319 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmdf4\"" Jul 09 02:15:00.147295 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.147272 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz"] Jul 09 02:15:00.235349 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.235308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98rtc\" (UniqueName: \"kubernetes.io/projected/63cc7283-d9fd-43cd-8b2b-f46852f178d1-kube-api-access-98rtc\") pod \"maas-api-key-cleanup-29726055-58wpz\" (UID: \"63cc7283-d9fd-43cd-8b2b-f46852f178d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" Jul 09 02:15:00.336320 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.336276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98rtc\" (UniqueName: \"kubernetes.io/projected/63cc7283-d9fd-43cd-8b2b-f46852f178d1-kube-api-access-98rtc\") pod \"maas-api-key-cleanup-29726055-58wpz\" (UID: \"63cc7283-d9fd-43cd-8b2b-f46852f178d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" Jul 09 02:15:00.345156 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.345120 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98rtc\" (UniqueName: \"kubernetes.io/projected/63cc7283-d9fd-43cd-8b2b-f46852f178d1-kube-api-access-98rtc\") pod \"maas-api-key-cleanup-29726055-58wpz\" (UID: \"63cc7283-d9fd-43cd-8b2b-f46852f178d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" Jul 09 02:15:00.450844 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.450746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" Jul 09 02:15:00.600721 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.599150 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz"] Jul 09 02:15:00.602706 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:15:00.602678 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cc7283_d9fd_43cd_8b2b_f46852f178d1.slice/crio-b37245d4e3aa8d7499f3dbb6c1e2cb4412afa8963e52d3261dba101e72569278 WatchSource:0}: Error finding container b37245d4e3aa8d7499f3dbb6c1e2cb4412afa8963e52d3261dba101e72569278: Status 404 returned error can't find the container with id b37245d4e3aa8d7499f3dbb6c1e2cb4412afa8963e52d3261dba101e72569278 Jul 09 02:15:00.752366 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:00.752332 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" event={"ID":"63cc7283-d9fd-43cd-8b2b-f46852f178d1","Type":"ContainerStarted","Data":"b37245d4e3aa8d7499f3dbb6c1e2cb4412afa8963e52d3261dba101e72569278"} Jul 09 02:15:03.770439 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:03.770398 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" event={"ID":"63cc7283-d9fd-43cd-8b2b-f46852f178d1","Type":"ContainerStarted","Data":"7bb721ec15f561d25216ff7889bd2c5fe3869e795f1f2fd05ed60d782609180f"} Jul 09 02:15:03.787557 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:15:03.787500 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" podStartSLOduration=1.166041955 podStartE2EDuration="3.787484251s" totalImagesPullingTime="2.621442296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:15:03.785930454 +0000 UTC m=+933.674093021" watchObservedRunningTime="2026-07-09 02:15:03.787484251 +0000 UTC m=+933.675646815" Jul 09 02:16:00.011243 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:00.011161 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz"] Jul 09 02:16:00.013704 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:00.011379 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" podUID="63cc7283-d9fd-43cd-8b2b-f46852f178d1" containerName="cleanup" containerID="cri-o://7bb721ec15f561d25216ff7889bd2c5fe3869e795f1f2fd05ed60d782609180f" gracePeriod=30 Jul 09 02:16:30.170803 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:30.170770 2578 generic.go:356] "Generic (PLEG): container finished" podID="63cc7283-d9fd-43cd-8b2b-f46852f178d1" containerID="7bb721ec15f561d25216ff7889bd2c5fe3869e795f1f2fd05ed60d782609180f" exitCode=137 Jul 09 02:16:30.171125 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:30.170799 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" event={"ID":"63cc7283-d9fd-43cd-8b2b-f46852f178d1","Type":"ContainerDied","Data":"7bb721ec15f561d25216ff7889bd2c5fe3869e795f1f2fd05ed60d782609180f"} Jul 09 02:16:30.189042 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:30.189020 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" Jul 09 02:16:30.263284 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:30.263229 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98rtc\" (UniqueName: \"kubernetes.io/projected/63cc7283-d9fd-43cd-8b2b-f46852f178d1-kube-api-access-98rtc\") pod \"63cc7283-d9fd-43cd-8b2b-f46852f178d1\" (UID: \"63cc7283-d9fd-43cd-8b2b-f46852f178d1\") " Jul 09 02:16:30.265453 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:30.265397 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cc7283-d9fd-43cd-8b2b-f46852f178d1-kube-api-access-98rtc" (OuterVolumeSpecName: "kube-api-access-98rtc") pod "63cc7283-d9fd-43cd-8b2b-f46852f178d1" (UID: "63cc7283-d9fd-43cd-8b2b-f46852f178d1"). InnerVolumeSpecName "kube-api-access-98rtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:16:30.364308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:30.364273 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98rtc\" (UniqueName: \"kubernetes.io/projected/63cc7283-d9fd-43cd-8b2b-f46852f178d1-kube-api-access-98rtc\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:16:31.176181 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:31.176150 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" Jul 09 02:16:31.176712 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:31.176185 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz" event={"ID":"63cc7283-d9fd-43cd-8b2b-f46852f178d1","Type":"ContainerDied","Data":"b37245d4e3aa8d7499f3dbb6c1e2cb4412afa8963e52d3261dba101e72569278"} Jul 09 02:16:31.176712 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:31.176242 2578 scope.go:117] "RemoveContainer" containerID="7bb721ec15f561d25216ff7889bd2c5fe3869e795f1f2fd05ed60d782609180f" Jul 09 02:16:31.192980 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:31.192952 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz"] Jul 09 02:16:31.196287 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:31.196265 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726055-58wpz"] Jul 09 02:16:32.579556 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:16:32.579516 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63cc7283-d9fd-43cd-8b2b-f46852f178d1" path="/var/lib/kubelet/pods/63cc7283-d9fd-43cd-8b2b-f46852f178d1/volumes" Jul 09 02:19:30.582085 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:19:30.582053 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:19:30.584827 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:19:30.584804 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:24:30.608926 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:24:30.608900 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:24:30.611873 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:24:30.611851 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:29:30.636267 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:29:30.636220 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:29:30.640538 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:29:30.640510 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:30:00.136413 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.136376 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp"] Jul 09 02:30:00.136910 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.136739 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63cc7283-d9fd-43cd-8b2b-f46852f178d1" containerName="cleanup" Jul 09 02:30:00.136910 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.136752 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="63cc7283-d9fd-43cd-8b2b-f46852f178d1" containerName="cleanup" Jul 09 02:30:00.136910 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.136806 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="63cc7283-d9fd-43cd-8b2b-f46852f178d1" containerName="cleanup" Jul 09 02:30:00.139789 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.139770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" Jul 09 02:30:00.142243 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.142223 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vmdf4\"" Jul 09 02:30:00.146170 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.146143 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp"] Jul 09 02:30:00.293402 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.293359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fggjs\" (UniqueName: \"kubernetes.io/projected/62ec6213-968b-45dd-bcba-08fd587bc247-kube-api-access-fggjs\") pod \"maas-api-key-cleanup-29726070-q8gtp\" (UID: \"62ec6213-968b-45dd-bcba-08fd587bc247\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" Jul 09 02:30:00.394808 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.394701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fggjs\" (UniqueName: \"kubernetes.io/projected/62ec6213-968b-45dd-bcba-08fd587bc247-kube-api-access-fggjs\") pod \"maas-api-key-cleanup-29726070-q8gtp\" (UID: \"62ec6213-968b-45dd-bcba-08fd587bc247\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" Jul 09 02:30:00.403241 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.403198 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fggjs\" (UniqueName: \"kubernetes.io/projected/62ec6213-968b-45dd-bcba-08fd587bc247-kube-api-access-fggjs\") pod \"maas-api-key-cleanup-29726070-q8gtp\" (UID: \"62ec6213-968b-45dd-bcba-08fd587bc247\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" Jul 09 02:30:00.452345 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.452299 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" Jul 09 02:30:00.608804 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.608778 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp"] Jul 09 02:30:00.611050 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:30:00.611021 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ec6213_968b_45dd_bcba_08fd587bc247.slice/crio-ec4116ca29577424b980bf384661d1efec04046b1e0252cde9d0829199d40cfe WatchSource:0}: Error finding container ec4116ca29577424b980bf384661d1efec04046b1e0252cde9d0829199d40cfe: Status 404 returned error can't find the container with id ec4116ca29577424b980bf384661d1efec04046b1e0252cde9d0829199d40cfe Jul 09 02:30:00.899010 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:00.898924 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" event={"ID":"62ec6213-968b-45dd-bcba-08fd587bc247","Type":"ContainerStarted","Data":"ec4116ca29577424b980bf384661d1efec04046b1e0252cde9d0829199d40cfe"} Jul 09 02:30:01.905803 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:01.905760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" event={"ID":"62ec6213-968b-45dd-bcba-08fd587bc247","Type":"ContainerStarted","Data":"a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b"} Jul 09 02:30:01.921208 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:30:01.921134 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" podStartSLOduration=1.9211127810000002 podStartE2EDuration="1.921112781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:30:01.920146294 +0000 UTC m=+1831.808308860" watchObservedRunningTime="2026-07-09 02:30:01.921112781 +0000 UTC m=+1831.809275347" Jul 09 02:31:00.010766 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:00.010687 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp"] Jul 09 02:31:00.011239 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:00.010914 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" podUID="62ec6213-968b-45dd-bcba-08fd587bc247" containerName="cleanup" containerID="cri-o://a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b" gracePeriod=30 Jul 09 02:31:21.387762 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.387730 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn"] Jul 09 02:31:21.391186 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.391164 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.393834 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.393810 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d099dd\"/\"openshift-service-ca.crt\"" Jul 09 02:31:21.394845 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.394815 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d099dd\"/\"kube-root-ca.crt\"" Jul 09 02:31:21.394967 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.394870 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d099dd\"/\"test-model-d099dd-kserve-self-signed-certs\"" Jul 09 02:31:21.395166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.395020 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d099dd\"/\"default-dockercfg-8m9s7\"" Jul 09 02:31:21.399153 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.399135 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn"] Jul 09 02:31:21.523430 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.523386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-dshm\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.523615 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.523435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-model-cache\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.523615 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.523496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-tmp-dir\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.523615 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.523516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212a2f87-7cc9-4e3c-b098-abc2497cf975-tls-certs\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.523615 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.523534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wbh\" (UniqueName: \"kubernetes.io/projected/212a2f87-7cc9-4e3c-b098-abc2497cf975-kube-api-access-t8wbh\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.523615 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.523604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-home\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.624967 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.624931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-tmp-dir\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.625111 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.624974 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212a2f87-7cc9-4e3c-b098-abc2497cf975-tls-certs\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.625156 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.625103 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wbh\" (UniqueName: \"kubernetes.io/projected/212a2f87-7cc9-4e3c-b098-abc2497cf975-kube-api-access-t8wbh\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.625156 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.625140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-home\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.625285 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.625261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-dshm\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.625360 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.625318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-model-cache\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.625454 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.625433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-tmp-dir\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.625580 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.625559 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-home\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.625701 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.625681 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-model-cache\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.627358 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.627337 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-dshm\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.627630 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.627614 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212a2f87-7cc9-4e3c-b098-abc2497cf975-tls-certs\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.633275 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.633233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wbh\" (UniqueName: \"kubernetes.io/projected/212a2f87-7cc9-4e3c-b098-abc2497cf975-kube-api-access-t8wbh\") pod \"test-model-d099dd-kserve-65f77d4fcc-6fdvn\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.701632 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.701530 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:21.833812 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:21.833787 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn"] Jul 09 02:31:21.835811 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:31:21.835784 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod212a2f87_7cc9_4e3c_b098_abc2497cf975.slice/crio-d988cde34fd2bd788936055ed4df2fde7e580dcbbea6c416e4cefcaafc2d5453 WatchSource:0}: Error finding container d988cde34fd2bd788936055ed4df2fde7e580dcbbea6c416e4cefcaafc2d5453: Status 404 returned error can't find the container with id d988cde34fd2bd788936055ed4df2fde7e580dcbbea6c416e4cefcaafc2d5453 Jul 09 02:31:22.287606 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:22.287569 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" event={"ID":"212a2f87-7cc9-4e3c-b098-abc2497cf975","Type":"ContainerStarted","Data":"323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233"} Jul 09 02:31:22.287606 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:22.287611 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" event={"ID":"212a2f87-7cc9-4e3c-b098-abc2497cf975","Type":"ContainerStarted","Data":"d988cde34fd2bd788936055ed4df2fde7e580dcbbea6c416e4cefcaafc2d5453"} Jul 09 02:31:22.304181 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:22.304104 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" podStartSLOduration=1.304084251 podStartE2EDuration="1.304084251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:31:22.302595901 +0000 UTC m=+1912.190758467" watchObservedRunningTime="2026-07-09 02:31:22.304084251 +0000 UTC m=+1912.192246818" Jul 09 02:31:30.195567 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.195527 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" Jul 09 02:31:30.309460 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.309362 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fggjs\" (UniqueName: \"kubernetes.io/projected/62ec6213-968b-45dd-bcba-08fd587bc247-kube-api-access-fggjs\") pod \"62ec6213-968b-45dd-bcba-08fd587bc247\" (UID: \"62ec6213-968b-45dd-bcba-08fd587bc247\") " Jul 09 02:31:30.311554 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.311509 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ec6213-968b-45dd-bcba-08fd587bc247-kube-api-access-fggjs" (OuterVolumeSpecName: "kube-api-access-fggjs") pod "62ec6213-968b-45dd-bcba-08fd587bc247" (UID: "62ec6213-968b-45dd-bcba-08fd587bc247"). InnerVolumeSpecName "kube-api-access-fggjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:31:30.326741 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.326709 2578 generic.go:356] "Generic (PLEG): container finished" podID="62ec6213-968b-45dd-bcba-08fd587bc247" containerID="a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b" exitCode=137 Jul 09 02:31:30.326893 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.326757 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" Jul 09 02:31:30.326893 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.326764 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" event={"ID":"62ec6213-968b-45dd-bcba-08fd587bc247","Type":"ContainerDied","Data":"a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b"} Jul 09 02:31:30.326893 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.326818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp" event={"ID":"62ec6213-968b-45dd-bcba-08fd587bc247","Type":"ContainerDied","Data":"ec4116ca29577424b980bf384661d1efec04046b1e0252cde9d0829199d40cfe"} Jul 09 02:31:30.326893 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.326841 2578 scope.go:117] "RemoveContainer" containerID="a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b" Jul 09 02:31:30.335361 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.335340 2578 scope.go:117] "RemoveContainer" containerID="a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b" Jul 09 02:31:30.335634 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:31:30.335611 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b\": container with ID starting with a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b not found: ID does not exist" containerID="a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b" Jul 09 02:31:30.335707 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.335640 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b"} err="failed to get container status \"a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b\": rpc error: code = NotFound desc = could not find container \"a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b\": container with ID starting with a4723c15861bb7348ffd14b105f8b8f8634849b41a7e836bd7e1b808f398555b not found: ID does not exist" Jul 09 02:31:30.347326 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.347295 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp"] Jul 09 02:31:30.352315 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.352288 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726070-q8gtp"] Jul 09 02:31:30.410433 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.410393 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fggjs\" (UniqueName: \"kubernetes.io/projected/62ec6213-968b-45dd-bcba-08fd587bc247-kube-api-access-fggjs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:31:30.580696 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:30.580617 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ec6213-968b-45dd-bcba-08fd587bc247" path="/var/lib/kubelet/pods/62ec6213-968b-45dd-bcba-08fd587bc247/volumes" Jul 09 02:31:31.702650 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:31.702614 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:31.703212 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:31.702923 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:31.715424 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:31.715397 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:32.349654 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:32.349621 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:31:39.277318 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.277280 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q"] Jul 09 02:31:39.277815 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.277794 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ec6213-968b-45dd-bcba-08fd587bc247" containerName="cleanup" Jul 09 02:31:39.277871 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.277819 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ec6213-968b-45dd-bcba-08fd587bc247" containerName="cleanup" Jul 09 02:31:39.277995 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.277981 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ec6213-968b-45dd-bcba-08fd587bc247" containerName="cleanup" Jul 09 02:31:39.281484 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.281462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.285147 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.285113 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a89bf\"/\"openshift-service-ca.crt\"" Jul 09 02:31:39.285147 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.285114 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a89bf\"/\"default-dockercfg-2nlf4\"" Jul 09 02:31:39.285147 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.285117 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5a89bf\"/\"test-model-5a89bf-kserve-self-signed-certs\"" Jul 09 02:31:39.285453 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.285151 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5a89bf\"/\"kube-root-ca.crt\"" Jul 09 02:31:39.290352 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.290321 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q"] Jul 09 02:31:39.390786 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.390749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67164c7b-86bb-4736-84b2-27623a5d03bf-tls-certs\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.390981 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.390788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-dshm\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.390981 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.390826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-model-cache\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.390981 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.390936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-home\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.390981 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.390975 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-tmp-dir\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.391147 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.390995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtg44\" (UniqueName: \"kubernetes.io/projected/67164c7b-86bb-4736-84b2-27623a5d03bf-kube-api-access-qtg44\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492131 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-home\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492343 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-tmp-dir\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492343 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtg44\" (UniqueName: \"kubernetes.io/projected/67164c7b-86bb-4736-84b2-27623a5d03bf-kube-api-access-qtg44\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492343 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67164c7b-86bb-4736-84b2-27623a5d03bf-tls-certs\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492343 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-dshm\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492343 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-model-cache\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492654 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-tmp-dir\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-home\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.492705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.492674 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-model-cache\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.494636 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.494615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-dshm\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.494814 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.494796 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67164c7b-86bb-4736-84b2-27623a5d03bf-tls-certs\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.500098 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.500075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtg44\" (UniqueName: \"kubernetes.io/projected/67164c7b-86bb-4736-84b2-27623a5d03bf-kube-api-access-qtg44\") pod \"test-model-5a89bf-kserve-69689c747-hn85q\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.592682 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.592578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:39.732037 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:39.732011 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q"] Jul 09 02:31:39.734097 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:31:39.734068 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67164c7b_86bb_4736_84b2_27623a5d03bf.slice/crio-00d15a2bbbc4e36130105701fd2e2b3c044830dfc19fc8097731c9ff680b2f36 WatchSource:0}: Error finding container 00d15a2bbbc4e36130105701fd2e2b3c044830dfc19fc8097731c9ff680b2f36: Status 404 returned error can't find the container with id 00d15a2bbbc4e36130105701fd2e2b3c044830dfc19fc8097731c9ff680b2f36 Jul 09 02:31:40.376008 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:40.375971 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" event={"ID":"67164c7b-86bb-4736-84b2-27623a5d03bf","Type":"ContainerStarted","Data":"52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429"} Jul 09 02:31:40.376430 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:40.376016 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" event={"ID":"67164c7b-86bb-4736-84b2-27623a5d03bf","Type":"ContainerStarted","Data":"00d15a2bbbc4e36130105701fd2e2b3c044830dfc19fc8097731c9ff680b2f36"} Jul 09 02:31:40.393101 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:40.393051 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" podStartSLOduration=1.393037025 podStartE2EDuration="1.393037025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:31:40.391787849 +0000 UTC m=+1930.279950413" watchObservedRunningTime="2026-07-09 02:31:40.393037025 +0000 UTC m=+1930.281199598" Jul 09 02:31:49.593024 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:49.592978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:49.593024 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:49.593035 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:49.605850 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:49.605824 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:31:50.434241 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:31:50.434213 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:32:05.677447 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:05.677415 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn"] Jul 09 02:32:06.615895 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:06.615842 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q"] Jul 09 02:32:20.697686 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.697643 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" podUID="212a2f87-7cc9-4e3c-b098-abc2497cf975" containerName="main" containerID="cri-o://323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233" gracePeriod=45 Jul 09 02:32:20.951283 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.951207 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:32:20.975843 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.975811 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-tmp-dir\") pod \"212a2f87-7cc9-4e3c-b098-abc2497cf975\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " Jul 09 02:32:20.976023 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.975869 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212a2f87-7cc9-4e3c-b098-abc2497cf975-tls-certs\") pod \"212a2f87-7cc9-4e3c-b098-abc2497cf975\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " Jul 09 02:32:20.976023 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.975963 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-home\") pod \"212a2f87-7cc9-4e3c-b098-abc2497cf975\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " Jul 09 02:32:20.976023 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.975995 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-model-cache\") pod \"212a2f87-7cc9-4e3c-b098-abc2497cf975\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " Jul 09 02:32:20.976023 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.976020 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8wbh\" (UniqueName: \"kubernetes.io/projected/212a2f87-7cc9-4e3c-b098-abc2497cf975-kube-api-access-t8wbh\") pod \"212a2f87-7cc9-4e3c-b098-abc2497cf975\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " Jul 09 02:32:20.976262 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.976030 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "212a2f87-7cc9-4e3c-b098-abc2497cf975" (UID: "212a2f87-7cc9-4e3c-b098-abc2497cf975"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:32:20.976262 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.976049 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-dshm\") pod \"212a2f87-7cc9-4e3c-b098-abc2497cf975\" (UID: \"212a2f87-7cc9-4e3c-b098-abc2497cf975\") " Jul 09 02:32:20.976374 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.976279 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-home" (OuterVolumeSpecName: "home") pod "212a2f87-7cc9-4e3c-b098-abc2497cf975" (UID: "212a2f87-7cc9-4e3c-b098-abc2497cf975"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:32:20.976374 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.976311 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-model-cache" (OuterVolumeSpecName: "model-cache") pod "212a2f87-7cc9-4e3c-b098-abc2497cf975" (UID: "212a2f87-7cc9-4e3c-b098-abc2497cf975"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:32:20.976477 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.976426 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-tmp-dir\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:20.976477 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.976447 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-home\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:20.978332 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.978301 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-dshm" (OuterVolumeSpecName: "dshm") pod "212a2f87-7cc9-4e3c-b098-abc2497cf975" (UID: "212a2f87-7cc9-4e3c-b098-abc2497cf975"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:32:20.978612 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.978584 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/212a2f87-7cc9-4e3c-b098-abc2497cf975-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "212a2f87-7cc9-4e3c-b098-abc2497cf975" (UID: "212a2f87-7cc9-4e3c-b098-abc2497cf975"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:32:20.978705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:20.978618 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212a2f87-7cc9-4e3c-b098-abc2497cf975-kube-api-access-t8wbh" (OuterVolumeSpecName: "kube-api-access-t8wbh") pod "212a2f87-7cc9-4e3c-b098-abc2497cf975" (UID: "212a2f87-7cc9-4e3c-b098-abc2497cf975"). InnerVolumeSpecName "kube-api-access-t8wbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:32:21.077266 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.077209 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-model-cache\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:21.077266 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.077239 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wbh\" (UniqueName: \"kubernetes.io/projected/212a2f87-7cc9-4e3c-b098-abc2497cf975-kube-api-access-t8wbh\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:21.077266 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.077270 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/212a2f87-7cc9-4e3c-b098-abc2497cf975-dshm\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:21.077513 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.077281 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/212a2f87-7cc9-4e3c-b098-abc2497cf975-tls-certs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:21.566891 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.566856 2578 generic.go:356] "Generic (PLEG): container finished" podID="212a2f87-7cc9-4e3c-b098-abc2497cf975" containerID="323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233" exitCode=0 Jul 09 02:32:21.567078 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.566899 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" Jul 09 02:32:21.567078 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.566943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" event={"ID":"212a2f87-7cc9-4e3c-b098-abc2497cf975","Type":"ContainerDied","Data":"323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233"} Jul 09 02:32:21.567078 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.566983 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn" event={"ID":"212a2f87-7cc9-4e3c-b098-abc2497cf975","Type":"ContainerDied","Data":"d988cde34fd2bd788936055ed4df2fde7e580dcbbea6c416e4cefcaafc2d5453"} Jul 09 02:32:21.567078 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.567002 2578 scope.go:117] "RemoveContainer" containerID="323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233" Jul 09 02:32:21.575967 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.575950 2578 scope.go:117] "RemoveContainer" containerID="323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233" Jul 09 02:32:21.576271 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:32:21.576234 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233\": container with ID starting with 323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233 not found: ID does not exist" containerID="323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233" Jul 09 02:32:21.576380 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.576281 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233"} err="failed to get container status \"323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233\": rpc error: code = NotFound desc = could not find container \"323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233\": container with ID starting with 323d3059a2246287af96a618e3db6dd17fa6550a8b576c37869f426f8ed94233 not found: ID does not exist" Jul 09 02:32:21.587771 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.587739 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn"] Jul 09 02:32:21.591322 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.591282 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d099dd/test-model-d099dd-kserve-65f77d4fcc-6fdvn"] Jul 09 02:32:21.640026 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.639981 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" podUID="67164c7b-86bb-4736-84b2-27623a5d03bf" containerName="main" containerID="cri-o://52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429" gracePeriod=45 Jul 09 02:32:21.889166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.889142 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:32:21.986522 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.986484 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-model-cache\") pod \"67164c7b-86bb-4736-84b2-27623a5d03bf\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " Jul 09 02:32:21.986522 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.986529 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-home\") pod \"67164c7b-86bb-4736-84b2-27623a5d03bf\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " Jul 09 02:32:21.986765 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.986555 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtg44\" (UniqueName: \"kubernetes.io/projected/67164c7b-86bb-4736-84b2-27623a5d03bf-kube-api-access-qtg44\") pod \"67164c7b-86bb-4736-84b2-27623a5d03bf\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " Jul 09 02:32:21.986765 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.986616 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-dshm\") pod \"67164c7b-86bb-4736-84b2-27623a5d03bf\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " Jul 09 02:32:21.986765 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.986665 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-tmp-dir\") pod \"67164c7b-86bb-4736-84b2-27623a5d03bf\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " Jul 09 02:32:21.986765 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.986698 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67164c7b-86bb-4736-84b2-27623a5d03bf-tls-certs\") pod \"67164c7b-86bb-4736-84b2-27623a5d03bf\" (UID: \"67164c7b-86bb-4736-84b2-27623a5d03bf\") " Jul 09 02:32:21.986961 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.986796 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "67164c7b-86bb-4736-84b2-27623a5d03bf" (UID: "67164c7b-86bb-4736-84b2-27623a5d03bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:32:21.986961 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.986896 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-home" (OuterVolumeSpecName: "home") pod "67164c7b-86bb-4736-84b2-27623a5d03bf" (UID: "67164c7b-86bb-4736-84b2-27623a5d03bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:32:21.987070 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.987008 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67164c7b-86bb-4736-84b2-27623a5d03bf" (UID: "67164c7b-86bb-4736-84b2-27623a5d03bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:32:21.987165 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.987073 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-model-cache\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:21.987165 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.987085 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-home\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:21.987165 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.987096 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-tmp-dir\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:21.988924 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.988904 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-dshm" (OuterVolumeSpecName: "dshm") pod "67164c7b-86bb-4736-84b2-27623a5d03bf" (UID: "67164c7b-86bb-4736-84b2-27623a5d03bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:32:21.989001 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.988963 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67164c7b-86bb-4736-84b2-27623a5d03bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67164c7b-86bb-4736-84b2-27623a5d03bf" (UID: "67164c7b-86bb-4736-84b2-27623a5d03bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:32:21.989042 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:21.989006 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67164c7b-86bb-4736-84b2-27623a5d03bf-kube-api-access-qtg44" (OuterVolumeSpecName: "kube-api-access-qtg44") pod "67164c7b-86bb-4736-84b2-27623a5d03bf" (UID: "67164c7b-86bb-4736-84b2-27623a5d03bf"). InnerVolumeSpecName "kube-api-access-qtg44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:32:22.088416 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.088366 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67164c7b-86bb-4736-84b2-27623a5d03bf-tls-certs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:22.088593 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.088427 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtg44\" (UniqueName: \"kubernetes.io/projected/67164c7b-86bb-4736-84b2-27623a5d03bf-kube-api-access-qtg44\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:22.088593 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.088437 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67164c7b-86bb-4736-84b2-27623a5d03bf-dshm\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:32:22.572359 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.572322 2578 generic.go:356] "Generic (PLEG): container finished" podID="67164c7b-86bb-4736-84b2-27623a5d03bf" containerID="52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429" exitCode=0 Jul 09 02:32:22.572544 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.572394 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" Jul 09 02:32:22.572544 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.572409 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" event={"ID":"67164c7b-86bb-4736-84b2-27623a5d03bf","Type":"ContainerDied","Data":"52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429"} Jul 09 02:32:22.572544 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.572455 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q" event={"ID":"67164c7b-86bb-4736-84b2-27623a5d03bf","Type":"ContainerDied","Data":"00d15a2bbbc4e36130105701fd2e2b3c044830dfc19fc8097731c9ff680b2f36"} Jul 09 02:32:22.572544 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.572481 2578 scope.go:117] "RemoveContainer" containerID="52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429" Jul 09 02:32:22.580301 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.580265 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="212a2f87-7cc9-4e3c-b098-abc2497cf975" path="/var/lib/kubelet/pods/212a2f87-7cc9-4e3c-b098-abc2497cf975/volumes" Jul 09 02:32:22.582186 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.582157 2578 scope.go:117] "RemoveContainer" containerID="52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429" Jul 09 02:32:22.582564 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:32:22.582543 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429\": container with ID starting with 52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429 not found: ID does not exist" containerID="52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429" Jul 09 02:32:22.582653 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.582574 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429"} err="failed to get container status \"52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429\": rpc error: code = NotFound desc = could not find container \"52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429\": container with ID starting with 52200e377533b47baedf6383a17779c4c78a89cee05eabbd8eb873c7bfc19429 not found: ID does not exist" Jul 09 02:32:22.595925 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.595895 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q"] Jul 09 02:32:22.599200 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:22.599174 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5a89bf/test-model-5a89bf-kserve-69689c747-hn85q"] Jul 09 02:32:24.580151 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:32:24.580115 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67164c7b-86bb-4736-84b2-27623a5d03bf" path="/var/lib/kubelet/pods/67164c7b-86bb-4736-84b2-27623a5d03bf/volumes" Jul 09 02:34:01.394603 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.394510 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp"] Jul 09 02:34:01.395120 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.394869 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="212a2f87-7cc9-4e3c-b098-abc2497cf975" containerName="main" Jul 09 02:34:01.395120 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.394884 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="212a2f87-7cc9-4e3c-b098-abc2497cf975" containerName="main" Jul 09 02:34:01.395120 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.394906 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67164c7b-86bb-4736-84b2-27623a5d03bf" containerName="main" Jul 09 02:34:01.395120 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.394912 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="67164c7b-86bb-4736-84b2-27623a5d03bf" containerName="main" Jul 09 02:34:01.395120 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.394972 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="212a2f87-7cc9-4e3c-b098-abc2497cf975" containerName="main" Jul 09 02:34:01.395120 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.394980 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="67164c7b-86bb-4736-84b2-27623a5d03bf" containerName="main" Jul 09 02:34:01.398038 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.398018 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.401689 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.401665 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7124ed\"/\"openshift-service-ca.crt\"" Jul 09 02:34:01.401821 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.401665 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7124ed\"/\"default-dockercfg-krmq7\"" Jul 09 02:34:01.402356 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.402337 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7124ed\"/\"kube-root-ca.crt\"" Jul 09 02:34:01.402503 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.402483 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7124ed\"/\"auth-test-model-7124ed-kserve-self-signed-certs\"" Jul 09 02:34:01.407131 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.407106 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp"] Jul 09 02:34:01.503369 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.503332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tmp-dir\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.503369 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.503371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-model-cache\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.503627 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.503386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tls-certs\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.503627 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.503504 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428lq\" (UniqueName: \"kubernetes.io/projected/3de9d265-96a1-4ad6-a7ec-2773a7509aae-kube-api-access-428lq\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.503627 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.503539 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-dshm\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.503627 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.503577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-home\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.604844 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.604801 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tmp-dir\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.604844 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.604842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-model-cache\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.604844 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.604857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tls-certs\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.605166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.604941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428lq\" (UniqueName: \"kubernetes.io/projected/3de9d265-96a1-4ad6-a7ec-2773a7509aae-kube-api-access-428lq\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.605166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.604971 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-dshm\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.605166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.605100 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-home\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.605365 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.605314 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tmp-dir\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.605420 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.605360 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-model-cache\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.605514 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.605491 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-home\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.607240 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.607208 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-dshm\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.607661 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.607642 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tls-certs\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.612296 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.612277 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-428lq\" (UniqueName: \"kubernetes.io/projected/3de9d265-96a1-4ad6-a7ec-2773a7509aae-kube-api-access-428lq\") pod \"auth-test-model-7124ed-kserve-79b6b8d475-2mzqp\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.708504 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.708473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:01.844120 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:01.844091 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp"] Jul 09 02:34:01.845907 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:34:01.845875 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3de9d265_96a1_4ad6_a7ec_2773a7509aae.slice/crio-12c0deef1a1e28cb44adae0fef65c48aea5cae9ecc972810976a2d5f0c0d4684 WatchSource:0}: Error finding container 12c0deef1a1e28cb44adae0fef65c48aea5cae9ecc972810976a2d5f0c0d4684: Status 404 returned error can't find the container with id 12c0deef1a1e28cb44adae0fef65c48aea5cae9ecc972810976a2d5f0c0d4684 Jul 09 02:34:02.029657 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:02.029567 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" event={"ID":"3de9d265-96a1-4ad6-a7ec-2773a7509aae","Type":"ContainerStarted","Data":"da70ac0bbcb5952f3a931c6cf3360642e2f2377a1b391003e7c9b6b780597e1a"} Jul 09 02:34:02.029657 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:02.029608 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" event={"ID":"3de9d265-96a1-4ad6-a7ec-2773a7509aae","Type":"ContainerStarted","Data":"12c0deef1a1e28cb44adae0fef65c48aea5cae9ecc972810976a2d5f0c0d4684"} Jul 09 02:34:02.047741 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:02.047673 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" podStartSLOduration=1.047657143 podStartE2EDuration="1.047657143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:34:02.046239288 +0000 UTC m=+2071.934401847" watchObservedRunningTime="2026-07-09 02:34:02.047657143 +0000 UTC m=+2071.935819752" Jul 09 02:34:11.709111 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:11.709069 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:11.709629 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:11.709125 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:11.721836 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:11.721812 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:12.090243 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:12.090160 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:34:28.890421 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.890383 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr"] Jul 09 02:34:28.894260 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.894226 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:28.897070 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.897048 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"openshift-service-ca.crt\"" Jul 09 02:34:28.898174 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.898149 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"kube-root-ca.crt\"" Jul 09 02:34:28.898456 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.898439 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"default-dockercfg-29rk9\"" Jul 09 02:34:28.898529 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.898450 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"auth-test-model-c6afad-kserve-self-signed-certs\"" Jul 09 02:34:28.904019 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.903993 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr"] Jul 09 02:34:28.943338 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.943296 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-tmp-dir\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:28.943338 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.943334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-dshm\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:28.943583 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.943367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-model-cache\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:28.943583 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.943460 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-home\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:28.943583 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.943488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcwr\" (UniqueName: \"kubernetes.io/projected/1eec180b-5720-4d8c-943a-6079d1dbfd97-kube-api-access-brcwr\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:28.943583 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:28.943535 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eec180b-5720-4d8c-943a-6079d1dbfd97-tls-certs\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.044375 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.044335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-home\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.044375 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.044373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcwr\" (UniqueName: \"kubernetes.io/projected/1eec180b-5720-4d8c-943a-6079d1dbfd97-kube-api-access-brcwr\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.044641 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.044478 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eec180b-5720-4d8c-943a-6079d1dbfd97-tls-certs\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.044641 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.044573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-tmp-dir\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.044641 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.044602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-dshm\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.044803 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.044667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-model-cache\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.044803 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.044774 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-home\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.045007 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.044981 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-tmp-dir\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.045073 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.045023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-model-cache\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.046804 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.046786 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-dshm\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.047054 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.047036 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eec180b-5720-4d8c-943a-6079d1dbfd97-tls-certs\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.052532 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.052510 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcwr\" (UniqueName: \"kubernetes.io/projected/1eec180b-5720-4d8c-943a-6079d1dbfd97-kube-api-access-brcwr\") pod \"auth-test-model-c6afad-kserve-79f6659559-9r6xr\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.206656 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.206624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:29.340496 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:29.340474 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr"] Jul 09 02:34:29.342769 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:34:29.342737 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1eec180b_5720_4d8c_943a_6079d1dbfd97.slice/crio-2a963adac21b0a770c7ce332f74ed10b886d94d1f9c9363bab1eab1ea327e696 WatchSource:0}: Error finding container 2a963adac21b0a770c7ce332f74ed10b886d94d1f9c9363bab1eab1ea327e696: Status 404 returned error can't find the container with id 2a963adac21b0a770c7ce332f74ed10b886d94d1f9c9363bab1eab1ea327e696 Jul 09 02:34:30.160501 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:30.160461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" event={"ID":"1eec180b-5720-4d8c-943a-6079d1dbfd97","Type":"ContainerStarted","Data":"9c28e97a83774d6e08b90e792ff8d96e324c84a1d2d407bd654a507a1762f226"} Jul 09 02:34:30.160501 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:30.160503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" event={"ID":"1eec180b-5720-4d8c-943a-6079d1dbfd97","Type":"ContainerStarted","Data":"2a963adac21b0a770c7ce332f74ed10b886d94d1f9c9363bab1eab1ea327e696"} Jul 09 02:34:30.182430 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:30.182370 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" podStartSLOduration=2.182352523 podStartE2EDuration="2.182352523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:34:30.181150189 +0000 UTC m=+2100.069312754" watchObservedRunningTime="2026-07-09 02:34:30.182352523 +0000 UTC m=+2100.070515087" Jul 09 02:34:30.663317 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:30.663290 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:34:30.669041 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:30.669014 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:34:39.207737 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:39.207704 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:39.208147 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:39.207776 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:39.220296 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:39.220271 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:34:40.218939 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:34:40.218911 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:36:27.534515 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:27.534470 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp"] Jul 09 02:36:28.580903 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:28.580873 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr"] Jul 09 02:36:42.554801 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.554742 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" podUID="3de9d265-96a1-4ad6-a7ec-2773a7509aae" containerName="main" containerID="cri-o://da70ac0bbcb5952f3a931c6cf3360642e2f2377a1b391003e7c9b6b780597e1a" gracePeriod=45 Jul 09 02:36:42.784900 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.784865 2578 generic.go:356] "Generic (PLEG): container finished" podID="3de9d265-96a1-4ad6-a7ec-2773a7509aae" containerID="da70ac0bbcb5952f3a931c6cf3360642e2f2377a1b391003e7c9b6b780597e1a" exitCode=0 Jul 09 02:36:42.785092 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.784943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" event={"ID":"3de9d265-96a1-4ad6-a7ec-2773a7509aae","Type":"ContainerDied","Data":"da70ac0bbcb5952f3a931c6cf3360642e2f2377a1b391003e7c9b6b780597e1a"} Jul 09 02:36:42.807524 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.807463 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:36:42.973232 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973198 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tmp-dir\") pod \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " Jul 09 02:36:42.973438 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973313 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-dshm\") pod \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " Jul 09 02:36:42.973438 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973382 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-home\") pod \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " Jul 09 02:36:42.973438 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973414 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-model-cache\") pod \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " Jul 09 02:36:42.973605 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973443 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-428lq\" (UniqueName: \"kubernetes.io/projected/3de9d265-96a1-4ad6-a7ec-2773a7509aae-kube-api-access-428lq\") pod \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " Jul 09 02:36:42.973605 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973483 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tls-certs\") pod \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\" (UID: \"3de9d265-96a1-4ad6-a7ec-2773a7509aae\") " Jul 09 02:36:42.973605 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973522 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3de9d265-96a1-4ad6-a7ec-2773a7509aae" (UID: "3de9d265-96a1-4ad6-a7ec-2773a7509aae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:36:42.973769 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973635 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-home" (OuterVolumeSpecName: "home") pod "3de9d265-96a1-4ad6-a7ec-2773a7509aae" (UID: "3de9d265-96a1-4ad6-a7ec-2773a7509aae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:36:42.973823 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973768 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-model-cache" (OuterVolumeSpecName: "model-cache") pod "3de9d265-96a1-4ad6-a7ec-2773a7509aae" (UID: "3de9d265-96a1-4ad6-a7ec-2773a7509aae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:36:42.973861 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973827 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-home\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:42.973861 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.973845 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tmp-dir\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:42.975573 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.975546 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-dshm" (OuterVolumeSpecName: "dshm") pod "3de9d265-96a1-4ad6-a7ec-2773a7509aae" (UID: "3de9d265-96a1-4ad6-a7ec-2773a7509aae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:36:42.975704 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.975659 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3de9d265-96a1-4ad6-a7ec-2773a7509aae" (UID: "3de9d265-96a1-4ad6-a7ec-2773a7509aae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:36:42.975813 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:42.975790 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de9d265-96a1-4ad6-a7ec-2773a7509aae-kube-api-access-428lq" (OuterVolumeSpecName: "kube-api-access-428lq") pod "3de9d265-96a1-4ad6-a7ec-2773a7509aae" (UID: "3de9d265-96a1-4ad6-a7ec-2773a7509aae"). InnerVolumeSpecName "kube-api-access-428lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:36:43.074624 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.074531 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-dshm\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:43.074624 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.074565 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3de9d265-96a1-4ad6-a7ec-2773a7509aae-model-cache\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:43.074624 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.074577 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-428lq\" (UniqueName: \"kubernetes.io/projected/3de9d265-96a1-4ad6-a7ec-2773a7509aae-kube-api-access-428lq\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:43.074624 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.074585 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3de9d265-96a1-4ad6-a7ec-2773a7509aae-tls-certs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:43.602349 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.602293 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" podUID="1eec180b-5720-4d8c-943a-6079d1dbfd97" containerName="main" containerID="cri-o://9c28e97a83774d6e08b90e792ff8d96e324c84a1d2d407bd654a507a1762f226" gracePeriod=45 Jul 09 02:36:43.792895 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.792869 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" Jul 09 02:36:43.793066 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.792861 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp" event={"ID":"3de9d265-96a1-4ad6-a7ec-2773a7509aae","Type":"ContainerDied","Data":"12c0deef1a1e28cb44adae0fef65c48aea5cae9ecc972810976a2d5f0c0d4684"} Jul 09 02:36:43.793066 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.793018 2578 scope.go:117] "RemoveContainer" containerID="da70ac0bbcb5952f3a931c6cf3360642e2f2377a1b391003e7c9b6b780597e1a" Jul 09 02:36:43.794767 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.794727 2578 generic.go:356] "Generic (PLEG): container finished" podID="1eec180b-5720-4d8c-943a-6079d1dbfd97" containerID="9c28e97a83774d6e08b90e792ff8d96e324c84a1d2d407bd654a507a1762f226" exitCode=0 Jul 09 02:36:43.794889 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.794794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" event={"ID":"1eec180b-5720-4d8c-943a-6079d1dbfd97","Type":"ContainerDied","Data":"9c28e97a83774d6e08b90e792ff8d96e324c84a1d2d407bd654a507a1762f226"} Jul 09 02:36:43.814201 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.814170 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp"] Jul 09 02:36:43.817123 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.817098 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/auth-test-model-7124ed-kserve-79b6b8d475-2mzqp"] Jul 09 02:36:43.858302 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.858215 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:36:43.981610 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.981576 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-tmp-dir\") pod \"1eec180b-5720-4d8c-943a-6079d1dbfd97\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " Jul 09 02:36:43.981610 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.981609 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-home\") pod \"1eec180b-5720-4d8c-943a-6079d1dbfd97\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " Jul 09 02:36:43.981870 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.981648 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-dshm\") pod \"1eec180b-5720-4d8c-943a-6079d1dbfd97\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " Jul 09 02:36:43.981870 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.981665 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eec180b-5720-4d8c-943a-6079d1dbfd97-tls-certs\") pod \"1eec180b-5720-4d8c-943a-6079d1dbfd97\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " Jul 09 02:36:43.981870 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.981689 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brcwr\" (UniqueName: \"kubernetes.io/projected/1eec180b-5720-4d8c-943a-6079d1dbfd97-kube-api-access-brcwr\") pod \"1eec180b-5720-4d8c-943a-6079d1dbfd97\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " Jul 09 02:36:43.981870 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.981742 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-model-cache\") pod \"1eec180b-5720-4d8c-943a-6079d1dbfd97\" (UID: \"1eec180b-5720-4d8c-943a-6079d1dbfd97\") " Jul 09 02:36:43.982071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.981952 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1eec180b-5720-4d8c-943a-6079d1dbfd97" (UID: "1eec180b-5720-4d8c-943a-6079d1dbfd97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:36:43.982071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.981968 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-home" (OuterVolumeSpecName: "home") pod "1eec180b-5720-4d8c-943a-6079d1dbfd97" (UID: "1eec180b-5720-4d8c-943a-6079d1dbfd97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:36:43.982169 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.982110 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-model-cache" (OuterVolumeSpecName: "model-cache") pod "1eec180b-5720-4d8c-943a-6079d1dbfd97" (UID: "1eec180b-5720-4d8c-943a-6079d1dbfd97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:36:43.982169 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.982152 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-tmp-dir\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:43.982277 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.982171 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-home\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:43.983947 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.983922 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-dshm" (OuterVolumeSpecName: "dshm") pod "1eec180b-5720-4d8c-943a-6079d1dbfd97" (UID: "1eec180b-5720-4d8c-943a-6079d1dbfd97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:36:43.984064 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.983980 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eec180b-5720-4d8c-943a-6079d1dbfd97-kube-api-access-brcwr" (OuterVolumeSpecName: "kube-api-access-brcwr") pod "1eec180b-5720-4d8c-943a-6079d1dbfd97" (UID: "1eec180b-5720-4d8c-943a-6079d1dbfd97"). InnerVolumeSpecName "kube-api-access-brcwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:36:43.984064 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:43.983991 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eec180b-5720-4d8c-943a-6079d1dbfd97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1eec180b-5720-4d8c-943a-6079d1dbfd97" (UID: "1eec180b-5720-4d8c-943a-6079d1dbfd97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:36:44.083019 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.082978 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-dshm\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:44.083019 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.083013 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eec180b-5720-4d8c-943a-6079d1dbfd97-tls-certs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:44.083019 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.083025 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brcwr\" (UniqueName: \"kubernetes.io/projected/1eec180b-5720-4d8c-943a-6079d1dbfd97-kube-api-access-brcwr\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:44.083290 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.083035 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eec180b-5720-4d8c-943a-6079d1dbfd97-model-cache\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:36:44.580483 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.580451 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3de9d265-96a1-4ad6-a7ec-2773a7509aae" path="/var/lib/kubelet/pods/3de9d265-96a1-4ad6-a7ec-2773a7509aae/volumes" Jul 09 02:36:44.801359 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.801330 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" Jul 09 02:36:44.801848 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.801362 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr" event={"ID":"1eec180b-5720-4d8c-943a-6079d1dbfd97","Type":"ContainerDied","Data":"2a963adac21b0a770c7ce332f74ed10b886d94d1f9c9363bab1eab1ea327e696"} Jul 09 02:36:44.801848 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.801406 2578 scope.go:117] "RemoveContainer" containerID="9c28e97a83774d6e08b90e792ff8d96e324c84a1d2d407bd654a507a1762f226" Jul 09 02:36:44.816852 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.816819 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr"] Jul 09 02:36:44.820555 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:44.820530 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/auth-test-model-c6afad-kserve-79f6659559-9r6xr"] Jul 09 02:36:46.580096 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:46.580056 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eec180b-5720-4d8c-943a-6079d1dbfd97" path="/var/lib/kubelet/pods/1eec180b-5720-4d8c-943a-6079d1dbfd97/volumes" Jul 09 02:36:56.587488 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.587407 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt"] Jul 09 02:36:56.587885 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.587806 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3de9d265-96a1-4ad6-a7ec-2773a7509aae" containerName="main" Jul 09 02:36:56.587885 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.587819 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3de9d265-96a1-4ad6-a7ec-2773a7509aae" containerName="main" Jul 09 02:36:56.587885 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.587828 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eec180b-5720-4d8c-943a-6079d1dbfd97" containerName="main" Jul 09 02:36:56.587885 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.587833 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eec180b-5720-4d8c-943a-6079d1dbfd97" containerName="main" Jul 09 02:36:56.588058 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.587891 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3de9d265-96a1-4ad6-a7ec-2773a7509aae" containerName="main" Jul 09 02:36:56.588058 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.587901 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eec180b-5720-4d8c-943a-6079d1dbfd97" containerName="main" Jul 09 02:36:56.592093 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.592074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.595809 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.595781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"default-dockercfg-29rk9\"" Jul 09 02:36:56.595916 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.595812 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"openshift-service-ca.crt\"" Jul 09 02:36:56.595916 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.595831 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"kube-root-ca.crt\"" Jul 09 02:36:56.595916 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.595783 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"sub-test-model-c6afad-kserve-self-signed-certs\"" Jul 09 02:36:56.598185 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.598156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tmp-dir\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.598354 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.598210 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-model-cache\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.598354 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.598326 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tls-certs\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.598484 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.598385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-dshm\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.598484 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.598412 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzkm\" (UniqueName: \"kubernetes.io/projected/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-kube-api-access-4lzkm\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.598484 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.598473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-home\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.599240 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.599219 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt"] Jul 09 02:36:56.699180 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699146 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tls-certs\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.699409 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-dshm\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.699409 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzkm\" (UniqueName: \"kubernetes.io/projected/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-kube-api-access-4lzkm\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.699409 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-home\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.699409 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tmp-dir\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.699409 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-model-cache\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.699768 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699748 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-home\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.699768 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699761 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-model-cache\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.699858 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.699808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tmp-dir\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.701487 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.701468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-dshm\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.701787 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.701771 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tls-certs\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.706875 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.706854 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzkm\" (UniqueName: \"kubernetes.io/projected/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-kube-api-access-4lzkm\") pod \"sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:56.902784 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:56.902687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:36:57.036872 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:57.036844 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt"] Jul 09 02:36:57.039106 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:36:57.039070 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a8d2b91_12d9_45ee_9f58_15b58aa3da9e.slice/crio-1968424768c7e6d59d985922c2e8e76c2878c7afc01fa084e98d022b7f642823 WatchSource:0}: Error finding container 1968424768c7e6d59d985922c2e8e76c2878c7afc01fa084e98d022b7f642823: Status 404 returned error can't find the container with id 1968424768c7e6d59d985922c2e8e76c2878c7afc01fa084e98d022b7f642823 Jul 09 02:36:57.861979 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:57.861941 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" event={"ID":"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e","Type":"ContainerStarted","Data":"c28f477d3aacb2e34b4eb28d7b97b41eaf819047a179eb55d16c82fdc817ecbc"} Jul 09 02:36:57.861979 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:57.861986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" event={"ID":"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e","Type":"ContainerStarted","Data":"1968424768c7e6d59d985922c2e8e76c2878c7afc01fa084e98d022b7f642823"} Jul 09 02:36:57.877735 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:36:57.877677 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" podStartSLOduration=1.8776556210000002 podStartE2EDuration="1.877655621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:36:57.876818805 +0000 UTC m=+2247.764981371" watchObservedRunningTime="2026-07-09 02:36:57.877655621 +0000 UTC m=+2247.765818189" Jul 09 02:37:06.903390 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:37:06.903361 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:37:06.904020 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:37:06.903406 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:37:06.916011 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:37:06.915986 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:37:07.919591 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:37:07.919559 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:37:48.000880 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:37:48.000842 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt"] Jul 09 02:38:03.021829 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.021759 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" podUID="2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" containerName="main" containerID="cri-o://c28f477d3aacb2e34b4eb28d7b97b41eaf819047a179eb55d16c82fdc817ecbc" gracePeriod=45 Jul 09 02:38:03.169427 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.169396 2578 generic.go:356] "Generic (PLEG): container finished" podID="2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" containerID="c28f477d3aacb2e34b4eb28d7b97b41eaf819047a179eb55d16c82fdc817ecbc" exitCode=0 Jul 09 02:38:03.169591 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.169469 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" event={"ID":"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e","Type":"ContainerDied","Data":"c28f477d3aacb2e34b4eb28d7b97b41eaf819047a179eb55d16c82fdc817ecbc"} Jul 09 02:38:03.271334 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.271307 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:38:03.406960 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.406862 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-dshm\") pod \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " Jul 09 02:38:03.406960 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.406918 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tls-certs\") pod \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " Jul 09 02:38:03.406960 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.406941 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tmp-dir\") pod \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " Jul 09 02:38:03.407283 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.407004 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-model-cache\") pod \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " Jul 09 02:38:03.407283 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.407025 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lzkm\" (UniqueName: \"kubernetes.io/projected/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-kube-api-access-4lzkm\") pod \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " Jul 09 02:38:03.407283 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.407088 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-home\") pod \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\" (UID: \"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e\") " Jul 09 02:38:03.407440 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.407280 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" (UID: "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:38:03.407440 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.407240 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" (UID: "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:38:03.407440 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.407419 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-home" (OuterVolumeSpecName: "home") pod "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" (UID: "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:38:03.407584 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.407454 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tmp-dir\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:38:03.407584 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.407468 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-model-cache\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:38:03.409126 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.409104 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" (UID: "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:38:03.409201 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.409150 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-kube-api-access-4lzkm" (OuterVolumeSpecName: "kube-api-access-4lzkm") pod "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" (UID: "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e"). InnerVolumeSpecName "kube-api-access-4lzkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:38:03.409201 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.409158 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-dshm" (OuterVolumeSpecName: "dshm") pod "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" (UID: "2a8d2b91-12d9-45ee-9f58-15b58aa3da9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:38:03.508068 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.508036 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lzkm\" (UniqueName: \"kubernetes.io/projected/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-kube-api-access-4lzkm\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:38:03.508068 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.508066 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-home\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:38:03.508301 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.508080 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-dshm\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:38:03.508301 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:03.508091 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e-tls-certs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:38:04.175433 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:04.175401 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" Jul 09 02:38:04.175433 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:04.175423 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt" event={"ID":"2a8d2b91-12d9-45ee-9f58-15b58aa3da9e","Type":"ContainerDied","Data":"1968424768c7e6d59d985922c2e8e76c2878c7afc01fa084e98d022b7f642823"} Jul 09 02:38:04.175988 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:04.175478 2578 scope.go:117] "RemoveContainer" containerID="c28f477d3aacb2e34b4eb28d7b97b41eaf819047a179eb55d16c82fdc817ecbc" Jul 09 02:38:04.196055 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:04.196024 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt"] Jul 09 02:38:04.197868 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:04.197845 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/sub-test-model-c6afad-kserve-8566f9f9bf-4zgqt"] Jul 09 02:38:04.580210 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:04.580172 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" path="/var/lib/kubelet/pods/2a8d2b91-12d9-45ee-9f58-15b58aa3da9e/volumes" Jul 09 02:38:10.083344 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.083306 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j"] Jul 09 02:38:10.083841 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.083756 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" containerName="main" Jul 09 02:38:10.083841 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.083771 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" containerName="main" Jul 09 02:38:10.083841 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.083837 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a8d2b91-12d9-45ee-9f58-15b58aa3da9e" containerName="main" Jul 09 02:38:10.086903 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.086879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.091052 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.091028 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7124ed\"/\"kube-root-ca.crt\"" Jul 09 02:38:10.091201 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.091057 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7124ed\"/\"openshift-service-ca.crt\"" Jul 09 02:38:10.091201 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.091030 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7124ed\"/\"default-dockercfg-krmq7\"" Jul 09 02:38:10.091201 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.091030 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7124ed\"/\"rate-test-model-7124ed-kserve-self-signed-certs\"" Jul 09 02:38:10.096097 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.096072 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j"] Jul 09 02:38:10.168459 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.168425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-dshm\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.168663 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.168469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e37236-abdb-4254-ac84-2f1d358a5979-tls-certs\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.168663 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.168523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-tmp-dir\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.168663 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.168586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-model-cache\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.168663 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.168643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-home\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.168846 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.168684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thr4f\" (UniqueName: \"kubernetes.io/projected/62e37236-abdb-4254-ac84-2f1d358a5979-kube-api-access-thr4f\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.269523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.269488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-tmp-dir\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.269523 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.269524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-model-cache\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.269889 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.269555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-home\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.269889 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.269590 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thr4f\" (UniqueName: \"kubernetes.io/projected/62e37236-abdb-4254-ac84-2f1d358a5979-kube-api-access-thr4f\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.269889 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.269625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-dshm\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.269889 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.269663 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e37236-abdb-4254-ac84-2f1d358a5979-tls-certs\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.270095 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.270045 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-home\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.270095 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.270072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-tmp-dir\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.270171 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.270097 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-model-cache\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.272135 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.272116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-dshm\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.272229 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.272212 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e37236-abdb-4254-ac84-2f1d358a5979-tls-certs\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.277274 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.277236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thr4f\" (UniqueName: \"kubernetes.io/projected/62e37236-abdb-4254-ac84-2f1d358a5979-kube-api-access-thr4f\") pod \"rate-test-model-7124ed-kserve-59d55d5567-s2c7j\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.397656 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.397545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:10.530285 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:10.530239 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j"] Jul 09 02:38:10.532401 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:38:10.532374 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62e37236_abdb_4254_ac84_2f1d358a5979.slice/crio-5200dea9b03230df49869aec829fe66d5baeb329508248550169aad3f1887732 WatchSource:0}: Error finding container 5200dea9b03230df49869aec829fe66d5baeb329508248550169aad3f1887732: Status 404 returned error can't find the container with id 5200dea9b03230df49869aec829fe66d5baeb329508248550169aad3f1887732 Jul 09 02:38:11.212204 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:11.212171 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" event={"ID":"62e37236-abdb-4254-ac84-2f1d358a5979","Type":"ContainerStarted","Data":"8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777"} Jul 09 02:38:11.212686 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:11.212217 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" event={"ID":"62e37236-abdb-4254-ac84-2f1d358a5979","Type":"ContainerStarted","Data":"5200dea9b03230df49869aec829fe66d5baeb329508248550169aad3f1887732"} Jul 09 02:38:11.231033 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:11.230981 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" podStartSLOduration=1.230963555 podStartE2EDuration="1.230963555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:38:11.228756239 +0000 UTC m=+2321.116918817" watchObservedRunningTime="2026-07-09 02:38:11.230963555 +0000 UTC m=+2321.119126120" Jul 09 02:38:20.398495 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:20.398449 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:20.398495 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:20.398508 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:20.411171 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:20.411135 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:21.267858 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:21.267826 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:38:52.695309 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.692843 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg"] Jul 09 02:38:52.700117 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.700092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.702121 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.702095 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg"] Jul 09 02:38:52.703006 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.702975 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"kube-root-ca.crt\"" Jul 09 02:38:52.704001 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.703980 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"default-dockercfg-29rk9\"" Jul 09 02:38:52.704277 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.704240 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"openshift-service-ca.crt\"" Jul 09 02:38:52.704371 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.704322 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c6afad\"/\"rate-test-model-c6afad-kserve-self-signed-certs\"" Jul 09 02:38:52.857903 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.857864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-model-cache\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.858118 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.857922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-dshm\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.858118 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.857953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-home\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.858118 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.857971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8qz\" (UniqueName: \"kubernetes.io/projected/17f7b283-a52e-46b0-925d-06e314a4c9ad-kube-api-access-sd8qz\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.858118 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.857995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-tmp-dir\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.858356 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.858160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17f7b283-a52e-46b0-925d-06e314a4c9ad-tls-certs\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959221 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17f7b283-a52e-46b0-925d-06e314a4c9ad-tls-certs\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959449 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-model-cache\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959449 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-dshm\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959449 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-home\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959449 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8qz\" (UniqueName: \"kubernetes.io/projected/17f7b283-a52e-46b0-925d-06e314a4c9ad-kube-api-access-sd8qz\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959449 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959408 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-tmp-dir\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959786 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959757 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-model-cache\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959847 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-home\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.959847 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.959833 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-tmp-dir\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.961645 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.961622 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-dshm\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.961895 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.961877 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17f7b283-a52e-46b0-925d-06e314a4c9ad-tls-certs\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:52.967678 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:52.967650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8qz\" (UniqueName: \"kubernetes.io/projected/17f7b283-a52e-46b0-925d-06e314a4c9ad-kube-api-access-sd8qz\") pod \"rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:53.010905 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:53.010867 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:38:53.148279 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:53.148232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg"] Jul 09 02:38:53.150941 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:38:53.150907 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17f7b283_a52e_46b0_925d_06e314a4c9ad.slice/crio-17e8bc59d32b55effa70167f15cbf48226b5befba4a674b698dc754f7912643c WatchSource:0}: Error finding container 17e8bc59d32b55effa70167f15cbf48226b5befba4a674b698dc754f7912643c: Status 404 returned error can't find the container with id 17e8bc59d32b55effa70167f15cbf48226b5befba4a674b698dc754f7912643c Jul 09 02:38:53.409924 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:53.409889 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" event={"ID":"17f7b283-a52e-46b0-925d-06e314a4c9ad","Type":"ContainerStarted","Data":"e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8"} Jul 09 02:38:53.409924 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:53.409933 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" event={"ID":"17f7b283-a52e-46b0-925d-06e314a4c9ad","Type":"ContainerStarted","Data":"17e8bc59d32b55effa70167f15cbf48226b5befba4a674b698dc754f7912643c"} Jul 09 02:38:53.426138 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:38:53.426077 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" podStartSLOduration=1.4260576710000001 podStartE2EDuration="1.426057671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 02:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:38:53.425006803 +0000 UTC m=+2363.313169368" watchObservedRunningTime="2026-07-09 02:38:53.426057671 +0000 UTC m=+2363.314220236" Jul 09 02:39:03.011502 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:03.011460 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:39:03.011937 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:03.011514 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:39:03.024151 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:03.024116 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:39:03.480028 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:03.476079 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:39:30.691939 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:30.691908 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:39:30.698373 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:30.698350 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:39:34.706969 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:34.706936 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j"] Jul 09 02:39:35.591546 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:35.591504 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg"] Jul 09 02:39:49.727693 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.727649 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" podUID="62e37236-abdb-4254-ac84-2f1d358a5979" containerName="main" containerID="cri-o://8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777" gracePeriod=45 Jul 09 02:39:49.976572 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.976544 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:39:49.997576 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.997410 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-model-cache\") pod \"62e37236-abdb-4254-ac84-2f1d358a5979\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " Jul 09 02:39:49.997576 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.997458 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-home\") pod \"62e37236-abdb-4254-ac84-2f1d358a5979\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " Jul 09 02:39:49.997576 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.997499 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-dshm\") pod \"62e37236-abdb-4254-ac84-2f1d358a5979\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " Jul 09 02:39:49.997576 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.997546 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thr4f\" (UniqueName: \"kubernetes.io/projected/62e37236-abdb-4254-ac84-2f1d358a5979-kube-api-access-thr4f\") pod \"62e37236-abdb-4254-ac84-2f1d358a5979\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " Jul 09 02:39:49.997935 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.997647 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-tmp-dir\") pod \"62e37236-abdb-4254-ac84-2f1d358a5979\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " Jul 09 02:39:49.997935 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.997683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e37236-abdb-4254-ac84-2f1d358a5979-tls-certs\") pod \"62e37236-abdb-4254-ac84-2f1d358a5979\" (UID: \"62e37236-abdb-4254-ac84-2f1d358a5979\") " Jul 09 02:39:49.997935 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.997733 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-model-cache" (OuterVolumeSpecName: "model-cache") pod "62e37236-abdb-4254-ac84-2f1d358a5979" (UID: "62e37236-abdb-4254-ac84-2f1d358a5979"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:39:49.998097 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.997993 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-model-cache\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:49.998436 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.998323 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-home" (OuterVolumeSpecName: "home") pod "62e37236-abdb-4254-ac84-2f1d358a5979" (UID: "62e37236-abdb-4254-ac84-2f1d358a5979"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:39:49.998436 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.998398 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62e37236-abdb-4254-ac84-2f1d358a5979" (UID: "62e37236-abdb-4254-ac84-2f1d358a5979"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:39:49.999981 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:49.999953 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-dshm" (OuterVolumeSpecName: "dshm") pod "62e37236-abdb-4254-ac84-2f1d358a5979" (UID: "62e37236-abdb-4254-ac84-2f1d358a5979"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:39:50.000071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.000033 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e37236-abdb-4254-ac84-2f1d358a5979-kube-api-access-thr4f" (OuterVolumeSpecName: "kube-api-access-thr4f") pod "62e37236-abdb-4254-ac84-2f1d358a5979" (UID: "62e37236-abdb-4254-ac84-2f1d358a5979"). InnerVolumeSpecName "kube-api-access-thr4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:39:50.000071 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.000045 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e37236-abdb-4254-ac84-2f1d358a5979-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62e37236-abdb-4254-ac84-2f1d358a5979" (UID: "62e37236-abdb-4254-ac84-2f1d358a5979"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:39:50.099492 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.099450 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-home\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:50.099492 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.099481 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-dshm\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:50.099492 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.099490 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thr4f\" (UniqueName: \"kubernetes.io/projected/62e37236-abdb-4254-ac84-2f1d358a5979-kube-api-access-thr4f\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:50.099492 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.099502 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62e37236-abdb-4254-ac84-2f1d358a5979-tmp-dir\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:50.099774 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.099510 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62e37236-abdb-4254-ac84-2f1d358a5979-tls-certs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:50.612954 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.612891 2578 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" podUID="17f7b283-a52e-46b0-925d-06e314a4c9ad" containerName="main" containerID="cri-o://e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8" gracePeriod=45 Jul 09 02:39:50.684017 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.683979 2578 generic.go:356] "Generic (PLEG): container finished" podID="62e37236-abdb-4254-ac84-2f1d358a5979" containerID="8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777" exitCode=0 Jul 09 02:39:50.684209 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.684047 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" Jul 09 02:39:50.684209 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.684076 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" event={"ID":"62e37236-abdb-4254-ac84-2f1d358a5979","Type":"ContainerDied","Data":"8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777"} Jul 09 02:39:50.684209 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.684124 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j" event={"ID":"62e37236-abdb-4254-ac84-2f1d358a5979","Type":"ContainerDied","Data":"5200dea9b03230df49869aec829fe66d5baeb329508248550169aad3f1887732"} Jul 09 02:39:50.684209 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.684141 2578 scope.go:117] "RemoveContainer" containerID="8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777" Jul 09 02:39:50.699800 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.699767 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j"] Jul 09 02:39:50.704832 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.704802 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7124ed/rate-test-model-7124ed-kserve-59d55d5567-s2c7j"] Jul 09 02:39:50.724171 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.724148 2578 scope.go:117] "RemoveContainer" containerID="8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777" Jul 09 02:39:50.724529 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:39:50.724508 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777\": container with ID starting with 8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777 not found: ID does not exist" containerID="8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777" Jul 09 02:39:50.724625 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.724538 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777"} err="failed to get container status \"8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777\": rpc error: code = NotFound desc = could not find container \"8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777\": container with ID starting with 8ee6637aa37a176e76271cf80edb77860b7769e3763d8eb88edaff7784b9d777 not found: ID does not exist" Jul 09 02:39:50.868527 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.868495 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:39:50.906290 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906224 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-home\") pod \"17f7b283-a52e-46b0-925d-06e314a4c9ad\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " Jul 09 02:39:50.906470 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906333 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd8qz\" (UniqueName: \"kubernetes.io/projected/17f7b283-a52e-46b0-925d-06e314a4c9ad-kube-api-access-sd8qz\") pod \"17f7b283-a52e-46b0-925d-06e314a4c9ad\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " Jul 09 02:39:50.906470 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906399 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-tmp-dir\") pod \"17f7b283-a52e-46b0-925d-06e314a4c9ad\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " Jul 09 02:39:50.906470 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906455 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-dshm\") pod \"17f7b283-a52e-46b0-925d-06e314a4c9ad\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " Jul 09 02:39:50.906630 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906497 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-model-cache\") pod \"17f7b283-a52e-46b0-925d-06e314a4c9ad\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " Jul 09 02:39:50.906630 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906449 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-home" (OuterVolumeSpecName: "home") pod "17f7b283-a52e-46b0-925d-06e314a4c9ad" (UID: "17f7b283-a52e-46b0-925d-06e314a4c9ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:39:50.906630 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906521 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17f7b283-a52e-46b0-925d-06e314a4c9ad-tls-certs\") pod \"17f7b283-a52e-46b0-925d-06e314a4c9ad\" (UID: \"17f7b283-a52e-46b0-925d-06e314a4c9ad\") " Jul 09 02:39:50.906630 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906589 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17f7b283-a52e-46b0-925d-06e314a4c9ad" (UID: "17f7b283-a52e-46b0-925d-06e314a4c9ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:39:50.906835 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906770 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "17f7b283-a52e-46b0-925d-06e314a4c9ad" (UID: "17f7b283-a52e-46b0-925d-06e314a4c9ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:39:50.906909 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906888 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-home\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:50.906965 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906911 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-tmp-dir\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:50.906965 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.906925 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-model-cache\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:50.908499 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.908480 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-dshm" (OuterVolumeSpecName: "dshm") pod "17f7b283-a52e-46b0-925d-06e314a4c9ad" (UID: "17f7b283-a52e-46b0-925d-06e314a4c9ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 02:39:50.908943 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.908911 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f7b283-a52e-46b0-925d-06e314a4c9ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17f7b283-a52e-46b0-925d-06e314a4c9ad" (UID: "17f7b283-a52e-46b0-925d-06e314a4c9ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 02:39:50.908943 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:50.908920 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f7b283-a52e-46b0-925d-06e314a4c9ad-kube-api-access-sd8qz" (OuterVolumeSpecName: "kube-api-access-sd8qz") pod "17f7b283-a52e-46b0-925d-06e314a4c9ad" (UID: "17f7b283-a52e-46b0-925d-06e314a4c9ad"). InnerVolumeSpecName "kube-api-access-sd8qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:39:51.007774 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.007730 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17f7b283-a52e-46b0-925d-06e314a4c9ad-dshm\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:51.007774 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.007766 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17f7b283-a52e-46b0-925d-06e314a4c9ad-tls-certs\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:51.007774 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.007777 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd8qz\" (UniqueName: \"kubernetes.io/projected/17f7b283-a52e-46b0-925d-06e314a4c9ad-kube-api-access-sd8qz\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:51.692206 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.692170 2578 generic.go:356] "Generic (PLEG): container finished" podID="17f7b283-a52e-46b0-925d-06e314a4c9ad" containerID="e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8" exitCode=0 Jul 09 02:39:51.692206 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.692207 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" Jul 09 02:39:51.692454 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.692236 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" event={"ID":"17f7b283-a52e-46b0-925d-06e314a4c9ad","Type":"ContainerDied","Data":"e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8"} Jul 09 02:39:51.692454 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.692289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg" event={"ID":"17f7b283-a52e-46b0-925d-06e314a4c9ad","Type":"ContainerDied","Data":"17e8bc59d32b55effa70167f15cbf48226b5befba4a674b698dc754f7912643c"} Jul 09 02:39:51.692454 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.692308 2578 scope.go:117] "RemoveContainer" containerID="e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8" Jul 09 02:39:51.701759 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.701733 2578 scope.go:117] "RemoveContainer" containerID="e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8" Jul 09 02:39:51.702047 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:39:51.702022 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8\": container with ID starting with e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8 not found: ID does not exist" containerID="e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8" Jul 09 02:39:51.702099 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.702062 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8"} err="failed to get container status \"e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8\": rpc error: code = NotFound desc = could not find container \"e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8\": container with ID starting with e224862bb2476738e7bd870490b1c39141f1e85f1ed83a9aee1461fd606546c8 not found: ID does not exist" Jul 09 02:39:51.712760 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.712728 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg"] Jul 09 02:39:51.715545 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.715520 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c6afad/rate-test-model-c6afad-kserve-6c7c89fddc-6xnvg"] Jul 09 02:39:51.915287 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.915176 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7362-140343326874480"] Jul 09 02:39:51.915672 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.915545 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e37236-abdb-4254-ac84-2f1d358a5979" containerName="main" Jul 09 02:39:51.915672 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.915557 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e37236-abdb-4254-ac84-2f1d358a5979" containerName="main" Jul 09 02:39:51.915672 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.915570 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f7b283-a52e-46b0-925d-06e314a4c9ad" containerName="main" Jul 09 02:39:51.915672 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.915576 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f7b283-a52e-46b0-925d-06e314a4c9ad" containerName="main" Jul 09 02:39:51.915672 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.915642 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e37236-abdb-4254-ac84-2f1d358a5979" containerName="main" Jul 09 02:39:51.915672 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.915655 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f7b283-a52e-46b0-925d-06e314a4c9ad" containerName="main" Jul 09 02:39:51.919752 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.919734 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7362-140343326874480" Jul 09 02:39:51.922649 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.922617 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-7tnpn\"" Jul 09 02:39:51.924286 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:51.924239 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7362-140343326874480"] Jul 09 02:39:52.016763 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.016726 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnczl\" (UniqueName: \"kubernetes.io/projected/24e2da01-a408-4d2a-a98b-321ad5c77f9c-kube-api-access-cnczl\") pod \"test-curl-7362-140343326874480\" (UID: \"24e2da01-a408-4d2a-a98b-321ad5c77f9c\") " pod="openshift-ingress/test-curl-7362-140343326874480" Jul 09 02:39:52.118197 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.118149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnczl\" (UniqueName: \"kubernetes.io/projected/24e2da01-a408-4d2a-a98b-321ad5c77f9c-kube-api-access-cnczl\") pod \"test-curl-7362-140343326874480\" (UID: \"24e2da01-a408-4d2a-a98b-321ad5c77f9c\") " pod="openshift-ingress/test-curl-7362-140343326874480" Jul 09 02:39:52.125899 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.125878 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnczl\" (UniqueName: \"kubernetes.io/projected/24e2da01-a408-4d2a-a98b-321ad5c77f9c-kube-api-access-cnczl\") pod \"test-curl-7362-140343326874480\" (UID: \"24e2da01-a408-4d2a-a98b-321ad5c77f9c\") " pod="openshift-ingress/test-curl-7362-140343326874480" Jul 09 02:39:52.230119 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.230083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7362-140343326874480" Jul 09 02:39:52.369272 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.369231 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7362-140343326874480"] Jul 09 02:39:52.371335 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:39:52.371307 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24e2da01_a408_4d2a_a98b_321ad5c77f9c.slice/crio-479ce592453c675dcff4f3ebc800c1fa40e9bec9c231aa8db8f7027ccbd95436 WatchSource:0}: Error finding container 479ce592453c675dcff4f3ebc800c1fa40e9bec9c231aa8db8f7027ccbd95436: Status 404 returned error can't find the container with id 479ce592453c675dcff4f3ebc800c1fa40e9bec9c231aa8db8f7027ccbd95436 Jul 09 02:39:52.372696 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.372674 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 02:39:52.581406 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.581322 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f7b283-a52e-46b0-925d-06e314a4c9ad" path="/var/lib/kubelet/pods/17f7b283-a52e-46b0-925d-06e314a4c9ad/volumes" Jul 09 02:39:52.581753 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.581737 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e37236-abdb-4254-ac84-2f1d358a5979" path="/var/lib/kubelet/pods/62e37236-abdb-4254-ac84-2f1d358a5979/volumes" Jul 09 02:39:52.700326 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:52.700285 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7362-140343326874480" event={"ID":"24e2da01-a408-4d2a-a98b-321ad5c77f9c","Type":"ContainerStarted","Data":"479ce592453c675dcff4f3ebc800c1fa40e9bec9c231aa8db8f7027ccbd95436"} Jul 09 02:39:53.705676 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:53.705639 2578 generic.go:356] "Generic (PLEG): container finished" podID="24e2da01-a408-4d2a-a98b-321ad5c77f9c" containerID="bfe60592ef778f177edeb93caae9dd317f488598bd17d221f8deaf77bfffb294" exitCode=0 Jul 09 02:39:53.706068 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:53.705720 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7362-140343326874480" event={"ID":"24e2da01-a408-4d2a-a98b-321ad5c77f9c","Type":"ContainerDied","Data":"bfe60592ef778f177edeb93caae9dd317f488598bd17d221f8deaf77bfffb294"} Jul 09 02:39:54.856173 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:54.856144 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7362-140343326874480" Jul 09 02:39:54.890603 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:54.890560 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7362-140343326874480_24e2da01-a408-4d2a-a98b-321ad5c77f9c/test-curl-7362-140343326874480/0.log" Jul 09 02:39:54.914862 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:54.914815 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7362-140343326874480"] Jul 09 02:39:54.918349 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:54.918320 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7362-140343326874480"] Jul 09 02:39:54.947036 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:54.947000 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnczl\" (UniqueName: \"kubernetes.io/projected/24e2da01-a408-4d2a-a98b-321ad5c77f9c-kube-api-access-cnczl\") pod \"24e2da01-a408-4d2a-a98b-321ad5c77f9c\" (UID: \"24e2da01-a408-4d2a-a98b-321ad5c77f9c\") " Jul 09 02:39:54.949293 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:54.949267 2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e2da01-a408-4d2a-a98b-321ad5c77f9c-kube-api-access-cnczl" (OuterVolumeSpecName: "kube-api-access-cnczl") pod "24e2da01-a408-4d2a-a98b-321ad5c77f9c" (UID: "24e2da01-a408-4d2a-a98b-321ad5c77f9c"). InnerVolumeSpecName "kube-api-access-cnczl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 02:39:55.047810 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:55.047697 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnczl\" (UniqueName: \"kubernetes.io/projected/24e2da01-a408-4d2a-a98b-321ad5c77f9c-kube-api-access-cnczl\") on node \"ip-10-0-131-83.ec2.internal\" DevicePath \"\"" Jul 09 02:39:55.718992 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:55.718962 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7362-140343326874480" Jul 09 02:39:55.719164 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:55.718969 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="479ce592453c675dcff4f3ebc800c1fa40e9bec9c231aa8db8f7027ccbd95436" Jul 09 02:39:55.730364 ip-10-0-131-83 kubenswrapper[2578]: E0709 02:39:55.730303 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7362-140343326874480\" is forbidden: User \"system:node:ip-10-0-131-83.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-131-83.ec2.internal' and this object" podUID="24e2da01-a408-4d2a-a98b-321ad5c77f9c" pod="openshift-ingress/test-curl-7362-140343326874480" Jul 09 02:39:56.579857 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:39:56.579820 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e2da01-a408-4d2a-a98b-321ad5c77f9c" path="/var/lib/kubelet/pods/24e2da01-a408-4d2a-a98b-321ad5c77f9c/volumes" Jul 09 02:40:09.643783 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:09.643743 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c5bd64b69-dspdm_5f5d397f-b8bd-452e-a4a2-3338c893dc2a/authorino/0.log" Jul 09 02:40:15.058963 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:15.058932 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8687897cbb-d6w6z_b64f0a10-3e72-475a-99c3-3e16d42fb9b6/copy-manifests/0.log" Jul 09 02:40:15.069344 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:15.069298 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8687897cbb-d6w6z_b64f0a10-3e72-475a-99c3-3e16d42fb9b6/manager/0.log" Jul 09 02:40:15.661437 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:15.661409 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-tkmzv_9d864215-dcdc-4079-bc73-907b09224c60/postgres/0.log" Jul 09 02:40:16.903461 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:16.903433 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c5bd64b69-dspdm_5f5d397f-b8bd-452e-a4a2-3338c893dc2a/authorino/0.log" Jul 09 02:40:17.230462 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:17.230430 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7mwx7_9d417a88-cff0-4635-93a7-24b03e16f478/kuadrant-console-plugin/0.log" Jul 09 02:40:18.388199 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:18.388162 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-847fc5468-fkkdf_b8d29b80-fcb4-4919-a72d-cbded5bc5f14/kube-auth-proxy/0.log" Jul 09 02:40:18.731154 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:18.731121 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-t2kp9_d3ce3fb6-b789-40a9-b139-6f0dbc840523/payload-processing/0.log" Jul 09 02:40:18.841801 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:18.841762 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b84dcd9-wrbcr_a98ef70b-a070-45fd-910e-0353efdd0278/router/0.log" Jul 09 02:40:19.176530 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:19.176451 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-268w6_a463e3c0-2ac4-41e1-8f83-45164d4e1702/main/0.log" Jul 09 02:40:19.396505 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:19.396479 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9ddr8_3146dcad-7a54-40d3-88d7-1cc853cb02c8/main/0.log" Jul 09 02:40:26.514936 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:26.514905 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fg9k4_c2f27e32-08c7-4607-9805-e280bcbcd8fe/global-pull-secret-syncer/0.log" Jul 09 02:40:26.680141 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:26.680095 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vkpgn_2ab7d58c-8222-4bb3-84b4-a09113c7fb8f/konnectivity-agent/0.log" Jul 09 02:40:26.749370 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:26.749334 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-83.ec2.internal_53be9f26ebbce9e8712878906521c6f8/haproxy/0.log" Jul 09 02:40:31.613563 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:31.613525 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c5bd64b69-dspdm_5f5d397f-b8bd-452e-a4a2-3338c893dc2a/authorino/0.log" Jul 09 02:40:31.732301 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:31.732270 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7mwx7_9d417a88-cff0-4635-93a7-24b03e16f478/kuadrant-console-plugin/0.log" Jul 09 02:40:33.678275 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:33.678223 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-72vjt_fb4e10fa-e512-499f-9cb3-9b98ed8bcba7/cluster-monitoring-operator/0.log" Jul 09 02:40:33.877643 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:33.877539 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2lzn6_7f77e720-6208-44b4-8eb9-b55ae2d4ce94/node-exporter/0.log" Jul 09 02:40:33.900271 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:33.900228 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2lzn6_7f77e720-6208-44b4-8eb9-b55ae2d4ce94/kube-rbac-proxy/0.log" Jul 09 02:40:33.922190 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:33.922158 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2lzn6_7f77e720-6208-44b4-8eb9-b55ae2d4ce94/init-textfile/0.log" Jul 09 02:40:34.212622 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:34.212584 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ee28e79-757c-438f-a450-afb20a69e324/prometheus/0.log" Jul 09 02:40:34.236923 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:34.236891 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ee28e79-757c-438f-a450-afb20a69e324/config-reloader/0.log" Jul 09 02:40:34.256430 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:34.256399 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ee28e79-757c-438f-a450-afb20a69e324/thanos-sidecar/0.log" Jul 09 02:40:34.277426 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:34.277376 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ee28e79-757c-438f-a450-afb20a69e324/kube-rbac-proxy-web/0.log" Jul 09 02:40:34.298221 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:34.298199 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ee28e79-757c-438f-a450-afb20a69e324/kube-rbac-proxy/0.log" Jul 09 02:40:34.319093 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:34.319052 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ee28e79-757c-438f-a450-afb20a69e324/kube-rbac-proxy-thanos/0.log" Jul 09 02:40:34.342075 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:34.342045 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7ee28e79-757c-438f-a450-afb20a69e324/init-config-reloader/0.log" Jul 09 02:40:35.925426 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:35.925393 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-rtsbr_369d323f-66e4-4b9a-a354-35926a34dfc3/networking-console-plugin/0.log" Jul 09 02:40:36.449152 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.449117 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/1.log" Jul 09 02:40:36.453642 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.453620 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-4mkwb_d84a9c83-d058-4182-814d-5e9a282d2013/console-operator/2.log" Jul 09 02:40:36.797822 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.797787 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992"] Jul 09 02:40:36.798147 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.798134 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e2da01-a408-4d2a-a98b-321ad5c77f9c" containerName="test-curl-7362-140343326874480" Jul 09 02:40:36.798195 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.798148 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e2da01-a408-4d2a-a98b-321ad5c77f9c" containerName="test-curl-7362-140343326874480" Jul 09 02:40:36.798231 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.798202 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e2da01-a408-4d2a-a98b-321ad5c77f9c" containerName="test-curl-7362-140343326874480" Jul 09 02:40:36.801165 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.801144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:36.803666 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.803646 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lshzb\"/\"openshift-service-ca.crt\"" Jul 09 02:40:36.803776 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.803668 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lshzb\"/\"default-dockercfg-mbz6n\"" Jul 09 02:40:36.804668 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.804654 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lshzb\"/\"kube-root-ca.crt\"" Jul 09 02:40:36.810089 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.810067 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992"] Jul 09 02:40:36.923985 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.923949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-sys\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:36.924166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.923995 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-proc\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:36.924166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.924087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-lib-modules\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:36.924166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.924111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m69lg\" (UniqueName: \"kubernetes.io/projected/48fa5100-51bb-4868-b64c-4e9109d04bc1-kube-api-access-m69lg\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:36.924166 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.924145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-podres\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:36.998352 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:36.998326 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-577b86448b-sr7t6_c9b33f23-24dc-4786-bb15-04a014e3de53/console/0.log" Jul 09 02:40:37.024705 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-proc\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.024916 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024742 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-lib-modules\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.024916 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69lg\" (UniqueName: \"kubernetes.io/projected/48fa5100-51bb-4868-b64c-4e9109d04bc1-kube-api-access-m69lg\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.024916 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024791 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-proc\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.024916 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-podres\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.024916 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-sys\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.025290 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024914 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-lib-modules\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.025290 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-sys\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.025290 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.024958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/48fa5100-51bb-4868-b64c-4e9109d04bc1-podres\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.025775 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.025760 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-9cmn7_92d4c717-baf8-4e62-b139-3e67293e89d5/download-server/0.log" Jul 09 02:40:37.032748 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.032725 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69lg\" (UniqueName: \"kubernetes.io/projected/48fa5100-51bb-4868-b64c-4e9109d04bc1-kube-api-access-m69lg\") pod \"perf-node-gather-daemonset-x4992\" (UID: \"48fa5100-51bb-4868-b64c-4e9109d04bc1\") " pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.111215 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.111114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.246418 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.246391 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992"] Jul 09 02:40:37.248808 ip-10-0-131-83 kubenswrapper[2578]: W0709 02:40:37.248780 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod48fa5100_51bb_4868_b64c_4e9109d04bc1.slice/crio-f7d2bfa3097ab79d6887a19a705fbf3a41c3842b1e1da09db1261e4d3fbe1294 WatchSource:0}: Error finding container f7d2bfa3097ab79d6887a19a705fbf3a41c3842b1e1da09db1261e4d3fbe1294: Status 404 returned error can't find the container with id f7d2bfa3097ab79d6887a19a705fbf3a41c3842b1e1da09db1261e4d3fbe1294 Jul 09 02:40:37.585360 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.585332 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-dg9nv_98d79bfc-4203-478c-bf3f-c6bac300e131/volume-data-source-validator/0.log" Jul 09 02:40:37.916390 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.916292 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" event={"ID":"48fa5100-51bb-4868-b64c-4e9109d04bc1","Type":"ContainerStarted","Data":"2e33eb0e103b885a8ab59fc355ff933f0ac4460439a48273fa4fcf16d73c1df0"} Jul 09 02:40:37.916540 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.916394 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:37.916540 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.916419 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" event={"ID":"48fa5100-51bb-4868-b64c-4e9109d04bc1","Type":"ContainerStarted","Data":"f7d2bfa3097ab79d6887a19a705fbf3a41c3842b1e1da09db1261e4d3fbe1294"} Jul 09 02:40:37.931920 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:37.931868 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" podStartSLOduration=1.931852669 podStartE2EDuration="1.931852669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 02:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 02:40:37.930334364 +0000 UTC m=+2467.818496923" watchObservedRunningTime="2026-07-09 02:40:37.931852669 +0000 UTC m=+2467.820015235" Jul 09 02:40:38.569750 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:38.569719 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-56q8v_50bc272c-2f83-44a1-9c62-6cbbd2e5a473/dns/0.log" Jul 09 02:40:38.590693 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:38.590665 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-56q8v_50bc272c-2f83-44a1-9c62-6cbbd2e5a473/kube-rbac-proxy/0.log" Jul 09 02:40:38.669636 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:38.669600 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hg8mp_e1f20951-58ef-4818-8715-c74ff5029e4d/dns-node-resolver/0.log" Jul 09 02:40:39.269579 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:39.269548 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w4w95_70db81f0-b646-4ae3-ab59-02cb9bc52828/node-ca/0.log" Jul 09 02:40:40.296874 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:40.296841 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-847fc5468-fkkdf_b8d29b80-fcb4-4919-a72d-cbded5bc5f14/kube-auth-proxy/0.log" Jul 09 02:40:40.432317 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:40.432286 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-t2kp9_d3ce3fb6-b789-40a9-b139-6f0dbc840523/payload-processing/0.log" Jul 09 02:40:40.450793 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:40.450763 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b84dcd9-wrbcr_a98ef70b-a070-45fd-910e-0353efdd0278/router/0.log" Jul 09 02:40:41.086507 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:41.086472 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2qgxg_abb47d9b-697a-4099-8423-03edd6d52ad6/serve-healthcheck-canary/0.log" Jul 09 02:40:41.568363 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:41.568326 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-n8hg2_13021762-a6ad-46ab-a4de-dc1838a6662e/insights-operator/0.log" Jul 09 02:40:41.570012 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:41.569997 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-n8hg2_13021762-a6ad-46ab-a4de-dc1838a6662e/insights-operator/1.log" Jul 09 02:40:41.648274 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:41.648236 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-msjzj_795ca69c-d52b-465e-a46d-454ced383b6b/kube-rbac-proxy/0.log" Jul 09 02:40:41.666325 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:41.666299 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-msjzj_795ca69c-d52b-465e-a46d-454ced383b6b/exporter/0.log" Jul 09 02:40:41.683911 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:41.683883 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-msjzj_795ca69c-d52b-465e-a46d-454ced383b6b/extractor/0.log" Jul 09 02:40:43.931375 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:43.931345 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lshzb/perf-node-gather-daemonset-x4992" Jul 09 02:40:44.413641 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:44.413612 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8687897cbb-d6w6z_b64f0a10-3e72-475a-99c3-3e16d42fb9b6/manager/0.log" Jul 09 02:40:44.430821 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:44.430795 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8687897cbb-d6w6z_b64f0a10-3e72-475a-99c3-3e16d42fb9b6/copy-manifests/0.log" Jul 09 02:40:45.680560 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:45.680527 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-596d76b798-wgcj8_c92a231d-0ae6-4881-953c-f7ce21aa6c54/manager/0.log" Jul 09 02:40:46.959308 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:46.959280 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-847fc5468-fkkdf_b8d29b80-fcb4-4919-a72d-cbded5bc5f14/kube-auth-proxy/0.log" Jul 09 02:40:47.071436 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:47.071352 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-t2kp9_d3ce3fb6-b789-40a9-b139-6f0dbc840523/payload-processing/0.log" Jul 09 02:40:47.086421 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:47.086399 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b84dcd9-wrbcr_a98ef70b-a070-45fd-910e-0353efdd0278/router/0.log" Jul 09 02:40:48.550663 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:48.550636 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-847fc5468-fkkdf_b8d29b80-fcb4-4919-a72d-cbded5bc5f14/kube-auth-proxy/0.log" Jul 09 02:40:48.661852 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:48.661815 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-t2kp9_d3ce3fb6-b789-40a9-b139-6f0dbc840523/payload-processing/0.log" Jul 09 02:40:48.676332 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:48.676240 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-789b84dcd9-wrbcr_a98ef70b-a070-45fd-910e-0353efdd0278/router/0.log" Jul 09 02:40:51.203926 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:51.203899 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-htm8p_71731d5b-3313-4046-8579-f828a13ba12c/kube-storage-version-migrator-operator/1.log" Jul 09 02:40:51.204780 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:51.204761 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-htm8p_71731d5b-3313-4046-8579-f828a13ba12c/kube-storage-version-migrator-operator/0.log" Jul 09 02:40:52.324332 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.324298 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqxdt_3c624a3e-0848-4b58-bb45-a1d1dbb59b8a/kube-multus-additional-cni-plugins/0.log" Jul 09 02:40:52.344006 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.343974 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqxdt_3c624a3e-0848-4b58-bb45-a1d1dbb59b8a/egress-router-binary-copy/0.log" Jul 09 02:40:52.362225 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.362195 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqxdt_3c624a3e-0848-4b58-bb45-a1d1dbb59b8a/cni-plugins/0.log" Jul 09 02:40:52.379208 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.379182 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqxdt_3c624a3e-0848-4b58-bb45-a1d1dbb59b8a/bond-cni-plugin/0.log" Jul 09 02:40:52.397062 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.397034 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqxdt_3c624a3e-0848-4b58-bb45-a1d1dbb59b8a/routeoverride-cni/0.log" Jul 09 02:40:52.414432 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.414403 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqxdt_3c624a3e-0848-4b58-bb45-a1d1dbb59b8a/whereabouts-cni-bincopy/0.log" Jul 09 02:40:52.438061 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.438029 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jqxdt_3c624a3e-0848-4b58-bb45-a1d1dbb59b8a/whereabouts-cni/0.log" Jul 09 02:40:52.597943 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.597860 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g2rkc_a6dd0c8d-d57a-428f-9803-3c570479d35e/kube-multus/0.log" Jul 09 02:40:52.697888 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.697854 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t7q22_c847f9eb-ebc3-40a4-a78c-e14cca47a814/network-metrics-daemon/0.log" Jul 09 02:40:52.714062 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:52.714039 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-t7q22_c847f9eb-ebc3-40a4-a78c-e14cca47a814/kube-rbac-proxy/0.log" Jul 09 02:40:53.519513 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:53.519485 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7cp8_1099b323-984e-47ae-9569-24c13d735e17/ovn-controller/0.log" Jul 09 02:40:53.535406 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:53.535373 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7cp8_1099b323-984e-47ae-9569-24c13d735e17/ovn-acl-logging/0.log" Jul 09 02:40:53.554256 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:53.554212 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7cp8_1099b323-984e-47ae-9569-24c13d735e17/kube-rbac-proxy-node/0.log" Jul 09 02:40:53.573878 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:53.573850 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7cp8_1099b323-984e-47ae-9569-24c13d735e17/kube-rbac-proxy-ovn-metrics/0.log" Jul 09 02:40:53.590074 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:53.590052 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7cp8_1099b323-984e-47ae-9569-24c13d735e17/northd/0.log" Jul 09 02:40:53.607674 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:53.607650 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7cp8_1099b323-984e-47ae-9569-24c13d735e17/nbdb/0.log" Jul 09 02:40:53.625759 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:53.625738 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7cp8_1099b323-984e-47ae-9569-24c13d735e17/sbdb/0.log" Jul 09 02:40:53.671472 ip-10-0-131-83 kubenswrapper[2578]: I0709 02:40:53.671438 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d7cp8_1099b323-984e-47ae-9569-24c13d735e17/ovnkube-controller/0.log"